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EXECUTIVE  SUMMARY 


The  U.S.  Army  Corps  of  Engineers  (USACE)  is  a  vital  component  of  U.S.  national 
defense  that  executes  necessary  construction  contracts  in  civil,  military,  and  contingency 
scenarios.  Near  the  epicenter  of  responsibility  are  contracting  officers  who  ensure  the 
USACE  mission  happens  with  the  greatest  of  efficiency.  Minimal  knowledge  of 
construction  contracting  is  imparted  to  an  Army  5 1C  contracting  officer  before  being 
assigned  to  the  USACE,  where  it  becomes  a  primary  function  of  the  contracting  officer. 
The  complexities  involved  in  contemporary  contracting  challenge  the  force  to  develop 
contracting  officers  capable  of  understanding  and  managing  contracts  to  efficiently  meet 
modem  demands.  In  2011,  the  USACE  produced  the  51C/1102  Proficiency  Guide 
(USACE,  2011)  in  an  effort  to  define  training  tasks  for  leaders  and  mentors  in  developing 
neophyte  contracting  officers  unfamiliar  with  construction  contracting. 

The  next  evolutionary  step  in  training  was  a  need  to  develop  a  supplemental  tool 
that  would  assist  mentors  in  assessing  competency  in  performing  the  45  tasks  in  the 
51C/1102  Proficiency  Guide  (USACE,  2011).  The  USACE  did  not  have  a  standardized 
test  to  measure  construction  contracting  skills.  The  use  of  a  proficiency  assessment  test 
(PAT)  by  the  USACE  would  provide  its  leaders  a  tool  to  mentor  5 1C  and  1102 
contracting  officers  through  immediate  feedback  on  the  proficiency  levels  of  their 
subordinates,  as  well  as  a  tangible  progress  tool  to  assist  them  in  providing  additional 
guidance.  This  PAT  tool  could  introduce  contracting  officers  new  to  the  USACE  with  the 
scope  of  USACE  operations  and  standards.  The  subject  matter  can  be  tested  to  identify 
tasks  that  the  individual  shows  proficiency  in  or  that  require  additional  attention.  Leaders 
could  then  individualize  or  focus  training  to  more  efficiently  address  the  needs  of 
subordinates. 

The  scope  of  the  research  team’s  project  involved  developing  a  proficiency 
assessment  test  that  covered  the  45  Contracting  tasks  listed  in  the  51C/1102  Proficiency 
Guide  (USACE,  2011).  Following  the  example  of  the  Arzu,  Castro,  &  Mack  research 
team  (Arzu,  2010)  that  developed  a  PAT  for  the  Army  Contracting  Command  (ACC),  the 
research  team  of  this  report  researched  and  created  a  similar  PAT  to  assist  the  USACE  in 


xv 


assessing  and  developing  the  proficiency  of  its  contracting  officers.  The  team  conducted 
a  literature  review  of  applicable  Army  training  doctrine  and  test-item  writing  to 
determine  methods  for  developing  the  PAT  and  writing  effective  questions.  The  team 
identified  appropriate  literature  to  become  familiar  with  construction  contracting, 
Architect-Engineer  contracting,  and  contingency  operations.  Because  of  the  specialized 
area  of  contracting  that  the  USACE  conducts,  the  preponderance  of  references  to 
understand  the  subject  matter  were  obtained  from  the  USACE  and  tied  to  its  published 
51  C/1 102  Proficiency  Guide  (USACE,  2011).  Utilizing  knowledge  gained  from  the 
literature  review,  the  team  developed  and  implemented  a  pilot  PAT  based  on  a  selected 
number  of  tasks.  Knowledgeable  MCCO  from  the  USACE  completed  this  pilot  PAT  and 
provided  feedback  on  the  developed  questions  and  format  that  were  incorporated  into  the 
development  of  the  comprehensive  PAT. 

The  research  team’s  test  items  were  vetted  by  USACE  subject  matter  experts  for 
realism,  difficulty,  and  time  required  for  completing  the  test.  Averaging  20  questions  per 
task,  the  research  team  provided  a  written  PAT  containing  approximately  800  test  items 
covering  the  45  tasks  identified  in  the  51C/1102  Proficiency  Guide  (USACE,  2011)  to  the 
USACE.  The  USACE  received  ownership  of  the  written  PAT  test  items  and  answers  that 
the  research  team  created.  The  USACE  can  relatively  easily  convert  the  PAT  from  its 
written  format  into  an  automated  version  for  implementation.  The  PAT  will  allow  leaders 
in  the  USACE  to  more  accurately  measure  a  contracting  officer’s  knowledge  and 
capabilities  against  the  established  standards  contained  in  the  51C/1102  Proficiency 
Guide. 
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I.  INTRODUCTION 


A.  BACKGROUND 

The  U.S.  Army  Corps  of  Engineers  (USACE)  has  played  an  integral  part  in  the 
development  of  the  country  and  its  changing  defense  requirements.  Over  the  last  two 
centuries,  the  duties  and  responsibilities  of  the  USACE  has  included  building  coastal 
fortifications,  surveying  roads  and  canals,  eliminating  navigational  hazards,  responding  to 
natural  disasters,  and  producing  both  military  construction  and  civil  works  (U.S.  Army 
Corps  of  Engineers  [USACE],  2012).  During  this  time  period,  procurement  of  contracted 
services  has  evolved  from  the  quartermaster  of  the  Revolutionary  War  to  today’s 
51C/1102  contracting  officers  (5 1C  is  the  military  occupational  specialty  [MOS]  code 
assigned  to  contracting  officers/specialists;  1102  is  the  civilian  equivalent  contracting 
officer/specialist).  The  complexities  involved  in  contemporary  contracting  challenge  the 
force  to  develop  contracting  officers  capable  of  understanding  and  managing  contracts  to 
efficiently  meet  modem  demands.  The  lack  of  a  standardized  training  regimen  with 
sufficient  preparation  and  mentorship  to  develop  contracting  officers  has  negatively 
affected  the  overall  performance  of  the  USACE  and  the  contracting  community  as  a 
whole. 

The  Army  established  the  Corps  of  Engineers  as  a  separate,  permanent  branch  on 
March  16,  1802.  Since  then,  the  USACE  has  acquired  an  exhaustive  list  of  agencies  and 
national  allies  as  service  clients.  It  manages  and/or  completes  contract  programs  at  home 
and  abroad  to  support  the  U.S.  Army  and  Air  Force,  federal  agencies  like  NASA  and  the 
U.S.  Postal  Service,  as  well  as  a  massive  effort  with  allies  in  Saudi  Arabia.  The  Corps 
also  maintains  a  rigorous  research  and  development  program  in  support  of  its  water 
resources,  construction,  and  military  activities.  Since  1941,  the  USACE  has  had  the 
responsibility  for  real  estate  acquisition,  construction,  and  maintenance  for  Army 
facilities,  including  training  camps,  government-owned  munitions  plants,  air  bases, 
depots,  and  hospitals.  Currently,  it  is  supporting  efforts  in  the  Global  War  on  Terrorism, 
including  reconstruction  in  Iraq  and  Afghanistan  (USACE,  2012). 
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This  broad  scope  of  operations  demands  knowledgeable  and  flexible  professionals 
to  competently  execute  the  Corps’  mission.  Due  to  the  systemic  failures  of  the 
contracting  profession  (Gansler  et  al.,  2007),  a  definitive  effort  to  improve  the  technical 
and  operational  capability  of  contracting  officers  is  being  demanded  across  the 
Department  of  Defense  (DoD).  Recommendations  by  the  Gansler  Commission, 
Department  of  Defense  Office  of  Inspector  General  (DoDIG),  and  others  mentioned  in 
the  literature  review  presented  in  Chapter  II,  detail  the  requirements  necessary  in 
overcoming  the  many  shortcomings  in  defense  contracting.  The  U.S.  Army  Corps  of 
Engineers  published  its  51C/1102  Proficiency  Guide  in  September  2011.  The  intent  of 
this  guide  is  not  to  replace  the  Army  Contracting  Command  5 1C  Level  One  Proficiency 
Guide  (Department  of  the  Army  [DoA],  2010)  but  to  supplement  it  for  construction  and 
architect-engineering  purposes.  The  51C/1102  Proficiency  Guide  identifies  essential 
tasks  expected  of  USACE  contracting  officers  and  provides  a  mentorship  tool  to  assist 
with  properly  developing  and  preparing  contracting  officers  to  meet  the  rigorous  needs  of 
USACE  operations. 

In  2010,  the  ACC  established  a  proficiency  assessment  test  (PAT)  for  the  36  tasks 
listed  in  the  Army  Contracting  Command  51C  Level  One  Proficiency  Guide  (DoA,  2010). 
Similarly,  this  research  utilizes  the  structure  and  order  of  the  45  tasks  listed  in  the 
USACE  51C/1102  Proficiency  Guide  for  Construction,  Architect-Engineer,  and 
Contingency  Contracting  (EP  715-1-8;  USACE,  2011a)  as  well  as  the  research  and 
methodologies  used  in  conceiving  the  ACC’s  PAT  to  create  a  separate  PAT  specifically 
for  construction  and  Architect-Engineer  contracting.  This  test  will  enhance  the  Army’s 
efforts  to  mentor  and  develop  5 1C  and  1102  contracting  officers  to  meet  the  USACE  and 
Army’s  standards  of  performance. 

B.  PROBLEM  STATEMENT 

The  recent  creation  and  distribution  of  the  51  C/1 102  Proficiency  Guide  for 
Construction,  Architect-Engineer  and  Contingency  Contracting  (EP  715-1-8;  USACE, 
2011a)  is  an  effective  tool  in  standardizing  the  vital  skills  in  the  51C/1102  repertoire.  The 
next  evolutionary  step  is  to  develop  a  supplemental  tool  that  will  assist  mentors  in 
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assessing  competency  in  performing  the  45  tasks  in  the  51  C/1 102  Proficiency  Guide. 
Currently,  the  USACE  does  not  have  a  standardized  test  to  measure  construction 
contracting  skills.  Construction  and  Architect-Engineer  contracting  are  areas  not 
addressed  in  the  ACC’s  PAT.  Creating  the  PAT  to  be  utilized  as  a  tool  will  assist  USACE 
mentors  in  measuring  the  proficiency  of  newer  5 1C  officers  and  1102  interns  against 
established  Army  standards  and  in  customizing  the  training  curriculum  to  more 
effectively  improve  the  progress  of  contracting  officers. 

C.  RESEARCH  OBJECTIVES 

The  objective  of  this  research  was  to  develop  a  written  PAT  for  the  45  identified 
tasks  in  the  USACE  51C/1102  Proficiency  Guide  (2011a).  The  research  team  developed 
questions  to  assess  proficiency  in  each  of  the  45  tasks  within  the  51C/1102  Proficiency 
Guide.  Utilizing  research  conducted  by  the  Arzu,  Castro,  and  Mack  (2010)  Naval 
Postgraduate  School  (NPS)  thesis  research  team,  applicable  theoretical  and  academic 
principles,  and  feedback  from  interviews  with  subject-matter  experts,  the  research  team 
developed  a  written  Construction  Contracting  Officer  Proficiency  Assessment  Test  to 
provide  to  the  USACE. 

To  accomplish  this,  the  research  team  answered  the  following  primary  research 
question:  What  is  the  most  effective  and  efficient  method  of  measuring  an  individual’s 
proficiency  in  the  45  identified  tasks  in  the  51C/1102  Proficiency  Guide  (USACE, 
2011a)?  Additionally,  the  team  researched  methods  to  develop  a  PAT  and  determined 
the  most  effective  type  of  test  for  evaluating  comprehension  and  recollection.  The  team 
also  implemented  a  pilot  PAT  with  the  assistance  of  its  targeted  audience,  USACE 
military  contingency  contracting  officers  (MCCOs),  to  identify  the  effectiveness  and 
limitations  of  the  research  team’s  approach  prior  to  finalizing  the  PAT. 

D.  SCOPE  AND  DELIVERABLES 

This  research  project’s  scope  is  limited  to  developing  a  proficiency  test  covering 
the  45  tasks  listed  in  the  51C/1102  Proficiency  Guide  (USACE,  2011a).  Completing  this 
research  project  required  the  team  to  execute  the  following  tasks: 
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•  develop  a  PAT  for  measuring  51C/1102  proficiency  at  the  tasks  listed  in 
the  51C/1102  Proficiency  Guide ,  which  included  questions  measuring 
comprehension  and  recollection  of  the  45  tasks; 

•  vet  questions  and  scenarios  through  feedback  from  USACE  subject-matter 
experts  to  confirm  the  validity  of  the  questions; 

•  construct  a  web-based  pilot  PAT  consisting  of  six  tasks  that 
knowledgeable  USACE  MCCOs  will  participate  in  to  confirm  that  test 
questions  meet  requested  time  and  difficulty  expectations  expressed  by  the 
USACE  and  its  test  proctors;  MCCO  feedback  will  be  incorporated  into 
the  PAT; 

•  complete  the  PAT  in  a  format  enabling  ease  of  use  in  either  written  or 
computerized  format;  and 

•  provide  a  compilation  of  written  questions  and  answers  to  the  USACE 
upon  completion  of  test  production. 

E.  POTENTIAL  BENEFITS 

The  use  of  a  proficiency  test  by  the  USACE  will  enable  its  leaders  to  mentor  5 1C 
and  1102  contracting  officers  to  have  immediate  feedback  on  the  proficiency  levels  of 
their  subordinates,  as  well  as  a  tangible  progress  tool  to  assist  them  in  providing 
additional  guidance.  This  tool  could  introduce  contracting  officers  new  to  the  USACE 
with  the  scope  of  USACE  operations  and  standards.  The  subject  matter  can  be  tested  to 
identify  tasks  that  the  individual  shows  proficiency  in  or  that  require  additional  attention. 
Leaders  could  then  individualize  or  focus  training  to  more  efficiently  address  the  needs  of 
subordinates.  The  instantaneous  feedback  from  the  test  would  also  lessen  the  time  needed 
by  commanders  to  assess  their  unit’s  training  effectiveness. 

The  PAT  will  also  quantify  the  51C/1102’s  preparedness  and  familiarity  with  the 
skills  necessary  to  accomplish  necessary  USACE  contracting  missions.  The  PAT  can  also 
serve  as  a  culminating  certification  of  an  individual’s  preparedness  after  completing  the 
unit’s  training  program.  Initial  attempts  by  the  individuals  can  be  recorded  and  compared 
with  later  efforts  to  demonstrate  progress  and  build  confidence  in  the  material. 

Additionally,  the  PAT  provides  vital  feedback  to  leaders  deciding  where  to  assign 
51Cs  and  1102s.  As  a  training  tool,  it  will  assist  leaders  in  identifying  individuals  who 
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are  prepared  for  promotion  to  positions  of  greater  responsibility.  This  could  improve  the 
overall  efficiency  of  the  unit  by  having  the  right  people,  competent  and  capable,  in  the 
right  positions. 

F.  LIMITATIONS 

The  establishment  and  implementation  of  the  PAT  has  limitations.  The  immediate 
limitations  are  the  delivery  method  to  test  individuals,  the  integrity  of  the  test,  and  the 
currency  of  the  PAT  based  on  the  currency  of  the  51C/1102  Proficiency  Guide ,  as  well  as 
updates  based  on  changes  in  policy  and  practice.  Additionally,  the  USACE  51C/1102 
Proficiency  Guide  (2011a)  tasks  have  varying  performance  measures  and  standards  that 
make  treating  testing  for  each  task  in  a  uniform  way  difficult.  Leadership  within  the 
USACE  will  need  to  implement  the  risk  management  process  to  mitigate  some  of  these 
limitations  to  ensure  the  PAT  is,  and  remains,  a  viable  and  effective  tool  for  assessment. 

The  delivery  method  to  test  individuals  will  have  distinctive  limitations  depending 
on  the  method  that  the  USACE  chooses.  The  primary  deliverable  for  this  research  is  the 
PAT  in  a  static  format  for  ease  of  delivery  in  either  written  or  computerized  form. 
Although  it  is  beyond  the  research  scope,  discussion  with  USACE  leadership  suggests 
that  an  online  delivery  method  will  be  strongly  considered  based  on  the  success  of  the 
Expeditionary  Contracting  Command’s  (ECC’s)  implementation.  The  limitations  of  this 
method  raise  several  questions  that  need  answers.  For  example,  who  will  be  in  charge  of 
implementing  the  test?  How  will  implementation  be  funded?  Will  the  test  be  proctored? 
How  often  will  it  be  administered?  The  differences  in  the  implementation  approach  will 
affect  the  resources  required  as  well  as  the  mitigation  measures  needed  to  protect  the 
integrity  of  the  PAT. 

The  integrity  of  the  PAT  is  a  concern.  Because  the  test  could  be  copied  and  the 
answers  disseminated  to  the  individuals  being  tested,  the  tool’s  purpose  and  effectiveness 
as  a  measurement  of  proficiency  could  be  invalidated.  Implementing  the  test  through  an 
automated  means  may  mitigate  some  of  the  risk  if  questions  are  selected  randomly  from  a 
database;  however,  the  threat  of  cheating  will  still  be  an  issue.  This  is  another  area  in 
which  USACE  leadership  can  mitigate  risk.  For  example,  personnel  being  assessed  could 
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be  required  to  take  the  PAT  at  a  proctored  location  with  sufficient  oversight  to  safeguard 
the  questions  and  prevent  answers  from  being  disseminated. 

Updates  to  policy  and  practice  are  another  limitation  to  the  research.  This  research 
is  based  on  the  current  51C/1102  Proficiency  Guide  (USACE,  2011a)  tasks  with  their 
associated  references  that  are  being  continuously  updated.  Therefore,  the  PAT  deliverable 
will  become  outdated  as  policies  and  practice  change  in  the  subject  areas.  The  USACE 
will  need  to  mitigate  this  limitation  by  updating  the  PAT  periodically  as  updates  are  made 
to  the  51C/1102  Proficiency  Guide  (USACE,  2011a)  to  ensure  that  the  PAT  remains  a 
viable  and  effective  tool  for  assessment. 

The  USACE  51C/1102  Proficiency  Guide  (2011a)  tasks  have  varying 
performance  measures  and  standards  that  make  treating  testing  for  each  task  in  a  uniform 
way  difficult.  As  is  discussed  later  in  Chapters  III  and  IV,  each  of  the  45  tasks  in  the 
USACE  51C/1102  Proficiency  Guide  (2011a)  have  different  performance  measures  and 
standards  that  define  acceptable  levels  of  proficiency  and  that  have  been  deconstructed  to 
formulate  questions  for  the  PAT.  Many  tasks  lend  themselves  to  a  uniform  approach  of 
utilizing  multiple-choice  questions;  however,  not  all  tasks  are  well  suited  for  a  multiple- 
choice  format.  This  limitation  needs  to  be  kept  in  mind  as  the  PAT  is  designed  as  a 
supplemental  tool  to  assist  mentors  in  assessing  proficiency.  The  PAT  should  not 
substitute  for  the  51C/1102  Proficiency  Guide  because  it  includes  several  tasks  that 
require  interactive  and  subjective  mentor  feedback  that  may  take  days  or  weeks  to 
accomplish  which  is  not  conducive  to  a  rapid  feedback  exam.  For  example,  Task  2-8, 
Develop  a  Construction  Solicitation,  is  best  assessed  by  measuring  the  quality  and 
thoroughness  of  a  developed  solicitation  and  not  by  answering  multiple-choice  questions 
that  include  the  correct  answers  (USACE,  2011a). 

Many  of  the  limitations  discussed  also  apply  to  the  ACC  PAT  initially  developed 
based  on  research  conducted  by  Arzu  et  al.  (2010).  Arzu  et  al.  (2010)  reached  similar 
conclusions  regarding  the  limitations  of  the  PAT.  The  ACC  was  able  to  adopt  mitigating 
practices,  such  as  automated  delivery  and  limited  distribution  of  the  answer  key,  that 
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have  protected  the  PAT  and  made  it  effective  and  efficient.  Similarly,  the  USACE  will 
need  to  determine  an  acceptable  level  of  risk  and  apply  risk  mitigation  techniques  to 
address  the  limitations  discussed. 

G.  METHODOLOGY 

The  research  methodology  was  to  first  grasp  a  clear  understanding  of  the 
requirement  needs  of  the  USACE,  the  sponsoring  agency,  to  determine  an  end-product 
deliverable  that  would  assist  the  agency  with  its  mission  needs.  Because  this  project 
closely  resembles  research  conducted  by  Arzu  et  al.  (2010),  the  team  became  familiar 
with  the  methodology  used  to  produce  an  effective  PAT  for  the  ACC  that  had  a  similar 
target  audience.  The  team  conducted  a  literature  review  of  applicable  Army  training 
doctrine  and  test-item  writing  to  determine  methods  for  developing  the  PAT  and  writing 
effective  questions.  The  team  identified  appropriate  literature  to  become  familiar  with 
construction  contracting,  Architect-Engineer  contracting,  and  contingency  operations. 
Because  of  the  specialized  area  of  contracting  that  the  USACE  conducts,  the 
preponderance  of  references  to  understand  the  subject  matter  were  obtained  from  the 
USACE  and  tied  to  its  published  51C/1102  Proficiency  Guide  (USACE,  2011a). 
Utilizing  knowledge  gained  from  the  literature  review,  the  team  developed  and 
implemented  a  pilot  PAT  based  on  a  limited  number  of  tasks.  Knowledgeable  MCCO 
from  the  USACE  completed  this  pilot  PAT  and  provided  feedback  on  the  developed 
questions  and  format.  The  team  incorporated  this  feedback  to  produce  PAT  questions 
covering  all  45  tasks  identified  in  the  51C/1102  Proficiency  Guide  (USACE,  2011a), 
which  were  then  reviewed  and  accepted  by  the  sponsoring  agency  for  future 
implementation. 

The  literature  review  of  USACE  contracting  subject  matter  is  extensive  in  both  its 
historical  timeline  and  breadth  of  subject  matter.  This  section  of  the  literature  review 
primarily  addresses  contemporary  issues  facing  ACC,  as  well  as  USACE-specific  topics 
in  the  21st  century.  Based  on  the  failures  identified  in  the  Gansler  Commission  report 
(Gansler  et  al.,  2007),  the  research  team  designated  this  document  as  a  starting  point  in 
the  literature  review.  The  USACE  Office  of  Inspector  General’s  database  provided 
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reports  applicable  to  the  research,  while  also  giving  the  explicit  details  of  common 
challenges  in  planning,  awarding,  and  administering  contracts.  The  Defense  Technical 
Information  Center  (DTIC)  contributed  additional  sources  on  topics  relevant  to 
construction  contracting  in  a  contingency  environment.  Supplementary  data  were 
acquired  through  reports  from  the  Government  Accountability  Office  (GAO),  USACE 
archives,  and  DoD  guidance.  Reviewing  USACE  history  and  identifying  the  contracting 
challenges  associated  with  the  applicable  contracting  mission  gave  the  research  team 
perspective  and  background  to  create  the  PAT. 

An  overview  of  the  USACE’s  National  Contracting  Organization  (NCO)  is 
discussed  to  ensure  that  the  research  in  developing  the  PAT  would  meet  NCO  mission 
needs.  The  NCO  overview  includes  the  organizational  structure,  command  relationship  to 
USACE,  and  training  methodology.  This  overview  includes  how  the  military  contingency 
contracting  teams  (MCCTs)  are  organized  and  the  approach  to  train  MCCOs  within  the 
NCO.  The  PAT  is  tailored  for  the  NCO-developed  51C/1102  Proficiency  Guide 
(USACE,  2011a),  which  is  an  essential  component  to  the  training  approach. 

The  51C/1102  Proficiency  Guide  for  Construction,  Architect-Engineer  and 
Contingency  Contracting  (USACE,  2011a)  is  the  primary  reference  for  PAT  question 
development.  The  51C/1102  Proficiency  Guide  has  45  specific  tasks  identified  that  are 
used  to  support  the  USACE  contracting  mission.  These  tasks  state  the  performance  steps 
required  to  meet  an  acceptable  standard  of  performance.  Using  the  51C/1102  Proficiency 
Guide  (USACE,  2011a)  was  crucial  to  the  research  team  in  building  the  physical  PAT,  as 
it  provided  an  easy  and  appropriate  format  for  commanders  to  assess  individual 
performance  against  overall  mission-essential  objectives.  The  performance  of  each  task, 
from  Task  1-1  through  Task  4-6,  is  measured  on  a  “GO”  or  “NO  GO”  basis,  which 
aligns  with  rating  methods  stated  in  Army  training  doctrine  (FM  7—1;  DoA,  2003).  The 
performance  steps  provided  the  key  focus  for  developing  answers  for  the  PAT.  The 
51C/1102  Proficiency  Guide  (USACE,  2011a)  also  links  references,  policies,  and 
procedures  to  the  tasks,  which  provided  the  research  team  with  sources  of  knowledge  to 
develop  and  validate  the  correctness  of  answers  for  the  PAT.  It  was  essential  for  the 
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research  team  to  become  familiar  with  the  51  C/1 102  Proficiency  Guide  (USACE,  2011a) 
and  how  it  supports  the  training  methodology  within  the  USACE. 

The  research  team  also  reviewed  Army  training  doctrine.  The  primary  audience 
for  taking  the  PAT  is  Army  soldiers  in  the  5 1C  career  field.  The  research  team  reviewed 
Army  Training  and  Leader  Development  (Army  Regulation  [AR]  350-1;  DoA,  2009), 
Training  the  Force  (Field  Manual  [FM]  7-0;  DoA,  2002),  and  Battle  Focused  Training 
(FM  7—1;  DoA,  2003)  because  they  provide  overarching  Army  doctrine  for  leader 
development  and  unit  training.  The  PAT  developed  in  this  project  can  be  successfully 
implemented  in  an  Army  center  or  school  as  an  established  method  of  preliminary 
training  and  successive  education.  Furthermore,  the  PAT  can  be  utilized  as  a  self¬ 
development  tool  throughout  the  professional  lifespan  of  an  Army  Soldier  or  professional 
civilian. 

One  of  the  challenges  of  the  project  was  integrating  theoretical  and  academic 
principles  with  Army  training  doctrine  in  a  manner  that  neither  diluted  nor  compromised 
either  component’s  fundamentals.  The  research  team  performed  a  literature  review  for  the 
development  and  construction  of  a  PAT.  The  research  team  utilized  Downing  and 
Haladyna’s  (2006)  Handbook  of  Test  Development  to  develop  a  test  construction  plan, 
test  blueprint  as  well  as  strategies  for  item  writing.  The  research  team  also  reviewed  other 
sources  such  as  The  Adult  Learner  (Knowles,  Holton,  &  Swanson,  1998)  for  learning 
theory  and  guidance  for  human  resource  development,  and  Test  Construction  (Wood, 
1960)  for  understanding  test  validity.  This  literature  provided  the  fundamentals  for  the 
test’s  justification.  Additionally,  the  research  team  reviewed  Manual  for  the  Preparation 
of  Objective  Test  Questions  (Muller,  2006)  to  further  understand  the  fundamentals  of  test- 
item  construction. 

Each  research  member  developed  questions  for  each  of  the  45  tasks  identified  by 

the  USACE  51C/1102  Proficiency  Guide  (2011a).  Having  each  member  look  at  each 

task,  rather  than  breaking  up  the  tasks  for  individual  effort,  allowed  each  research  team 

member  to  be  familiar  with  all  the  references,  enabled  the  research  team  to  brainstorm  as 

a  group,  and  allowed  team  members  to  peer  review  questions  other  team  members  put 

forward.  As  mentioned,  the  51C/1102  Proficiency  Guide  (2011a)  follows  Army  training 
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doctrine  by  identifying  essential  tasks  to  train  and  gives  conditions,  standards,  and 
performance  steps  to  achieve  the  acceptable  level  of  performance  for  each  task.  The 
research  team’s  approach  was  to  “reverse-engineer”  the  tasks,  using  an  approach  similar 
to  that  used  by  Arzu  et  al.  (2010),  by  identifying  the  attributes  of  demonstrated 
proficiency  and  making  those  responses  the  answers  to  focus  question  development 
around  for  the  PAT. 

Utilizing  knowledge  gained  from  the  literature  review,  the  team  developed  and 
implemented  a  pilot  PAT  consisting  of  six  tasks  that  knowledgeable  USACE  MCCOs 
participated  in.  The  MCCOs  provided  feedback  to  the  team  on  the  developed  questions 
and  PAT  format.  The  team  incorporated  the  feedback  received  to  continue  producing 
PAT  questions  covering  all  45  tasks  identified  in  the  51C/1102  Proficiency  Guide 
(USACE,  2011a).  As  groups  of  tasks  were  completed  by  subject  area,  the  team  provided 
the  research  to  the  sponsor  point  of  contact  within  the  USACE.  The  USACE  utilized 
knowledgeable  and  experienced  leaders  within  their  organization  to  review  the  material 
and  provide  feedback. 

H.  ORGANIZATION  OF  THIS  REPORT 

The  organization  of  this  report  consists  of  five  chapters.  Chapter  I  is  an 
introduction  to  the  research  project.  The  introduction  consists  of  the  following  topics: 
background  information  on  the  problem,  the  research  problem  statement,  research 
objectives,  the  project  scope  and  deliverables  to  the  USACE,  potential  benefits  of 
developing  a  PAT,  limitations  impacting  the  PAT,  the  research  methodology, 
organization  of  the  report,  and  a  summary.  Chapter  II  is  a  literature  review  consisting  of  a 
multitude  of  sources  on  the  following  topics:  common  contracting  challenges  and  issues 
applicable  to  USACE  contracting,  Army  training  doctrine  applicable  to  the  problem,  and 
test  construction  and  evaluation.  In  Chapter  III,  the  researchers  give  an  overview  of  the 
USACE  NCO  as  the  developing  agency  for  the  51C/1102  Proficiency  Guide  (USACE, 
2011a),  which  is  discussed  in  detail.  In  Chapter  IV,  the  researchers  describe  the  research 
approach  and  method  to  create  the  PAT  as  well  as  the  production  of  and  feedback  from 
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the  pilot  PAT.  In  Chapter  V,  the  researchers  summarize  the  project  and  present  the 
conclusion  and  recommendation  areas  for  further  research. 

I.  SUMMARY 

The  USACE  has  played  an  integral  part  in  the  development  of  the  country 
and  its  changing  defense  requirements.  Over  the  last  two  centuries,  the  duties  and 
responsibilities  of  the  USACE  have  included  building  coastal  fortifications,  surveying 
roads  and  canals,  eliminating  navigational  hazards,  responding  to  natural  disasters,  and 
producing  both  military  construction  and  civil  works  (USACE,  2012).  The  lack  of  a 
standardized  training  regimen  with  sufficient  preparation  and  mentorship  of  developing 
the  USACE  acquisition  workforce  has  negatively  affected  the  overall  performance  of  the 
USACE  and  the  contracting  community  as  a  whole.  The  USACE  published  its  51C/1102 
Proficiency  Guide  in  September  2011.  The  51  C/1 102  Proficiency  Guide  identifies 
essential  tasks  expected  of  USACE  contracting  officers  and  provides  a  mentorship  tool  to 
assist  with  properly  developing  and  preparing  contracting  officers  to  meet  the  rigorous 
needs  of  USACE  operations. 

The  recent  creation  and  distribution  of  the  USACE  51C/1102  Proficiency  Guide 
(2011a)  is  an  effective  tool  in  standardizing  the  vital  skills  in  the  51C/1102  repertoire. 
The  next  evolutionary  step  is  to  develop  a  supplemental  tool  that  will  assist  mentors  in 
assessing  competency  in  performing  the  45  tasks  in  the  51C/1102  Proficiency  Guide 
(USACE,  2011a).  The  objective  of  this  research  is  to  develop  a  written  PAT  for  the 
45  identified  tasks  in  the  USACE  51C/1102  Proficiency  Guide  (2011a).  A  proficiency 
test  will  enable  USACE  leaders  mentoring  5 1C  and  1102  contracting  officers  to  have 
immediate  feedback  on  the  proficiency  levels  of  subordinates  as  well  as  a  tangible 
progress  tool  that  will  assist  them  in  providing  additional  guidance.  This  tool  could 
introduce  contracting  officers  new  to  the  USACE  with  the  scope  of  USACE  operations 
and  standards.  Additionally,  the  PAT  can  quantify  USACE  team  members’  preparedness 
and  familiarity  with  the  skills  necessary  to  accomplish  necessary  USACE  contracting 
missions  and  may  provide  USACE  leadership  with  decision-making  tools  to  decide 
where  to  assign  51Cs  and  1102s. 
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The  establishment  and  implementation  of  the  PAT  has  limitations.  The  immediate 
limitations  are  the  delivery  method  to  test  individuals,  the  integrity  of  the  test,  and 
updates  based  on  changes  in  policy  and  practice.  Additionally,  the  USACE  51C/1102 
Proficiency  Guide  (2011a)  tasks  themselves  have  varying  performance  measures  and 
standards  that  make  treating  testing  for  each  task  in  a  uniform  way  difficult.  Leadership 
within  the  USACE  will  need  to  implement  the  risk  management  process  to  mitigate  some 
of  these  limitations  and  ensure  the  PAT  is,  and  remains,  a  viable  and  effective  tool  for 
assessment. 

Establishing  a  PAT  based  on  the  51C/1102  Proficiency  Guide  (USACE,  2011a) 
will  strengthen  the  USACE’s  training  methodology.  The  PAT  will  provide  an  additional 
tool  for  the  contracting  workforce  to  demonstrate  task  proficiency  to  themselves,  their 
supervisors,  and  other  leaders.  Creating  the  PAT  as  a  supplementary  tool  to  the  51C/1102 
Proficiency  Guide  will  further  assist  USACE  mentors  in  measuring  the  proficiency  of 
newer  5 1C  officers  and  1102  interns  against  established  Army  standards  and  in 
customizing  their  training  curriculum  to  more  effectively  improve  contracting  officers’ 
progress. 

Chapter  II  is  the  preponderance  of  the  research  literature  review.  The  review 
primarily  addresses  contemporary  issues  facing  ACC — as  well  as  USACE-specific  topics 
in  the  21st  century.  The  review  probes  the  problems  and  challenges  of  contracting  with  a 
focus  on  construction,  Architect-Engineer,  and  contingency  contracting.  Additionally, 
applicable  Army  training  doctrine  is  discussed  as  well  as  a  review  of  the  literature  on  test 
construction  and  evaluation. 
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II.  LITERATURE  REVIEW 


A.  INTRODUCTION 

In  June  2010,  the  Naval  Postgraduate  School  project  team  of  MAJ  Juan  Arzu, 
MAJ  Beire  Castro,  and  MAJ  Brian  Mack  produced  an  MBA  professional  report, 
Contingency  Contracting  Officer  Proficiency  Assessment  Test  Development .  Arzu  et  al. 
addressed  the  need  for  an  effective  and  efficient  method  of  measuring  contracting 
officers’  proficiency  at  assigned  tasks  and  asked  how  the  test  can  be  developed  to  be  best 
suited  for  meeting  their  client’s  requirements.  Their  study  reviewed  two  major 
components:  contingency  contracting  and  test  construction  and  evaluation.  By 
researching  and  understanding  the  first,  Arzu  et  al.  (2010)  defined  requirements,  scope, 
and  depth  for  the  second  component. 

The  field  of  test  development  has  not  made  many  radical  changes  or 
improvements  from  what  was  created  50  years  ago.  This  lack  of  change  is  evidenced  by 
the  many  contemporary  studies  that  cite  Bloom’s  Taxonomy  of  Educational  Objectives 
(1956)  and  Nunnally  and  Ator’s  Educational  Measurement  and  Evaluation  (1972),  often 
using  these  works  as  the  basis  of  the  studies.  The  Arzu  et  al.  (2010)  research  team 
structured  its  questions  according  to  Bloom’s  cognitive  domains  of  remembering, 
understanding,  and  applying  to  tasks,  and  categorized  their  questions  according  to 
Nunnally  and  Ator’s  (1972)  principles.  The  team  also  implemented  further  refinements 
based  on  Anderson  et  al.’s  A  Taxonomy  for  Learning,  Teaching,  and  Assessing  (2002), 
which  was  a  revision  and  extension  of  Bloom’s  (1956)  earlier  work.  The  Arzu  et  al. 
(2010)  professional  report  not  only  guided  the  research  team  of  this  report  through  the 
rigors  of  developing  a  method  for  measuring  individual  competencies  of  an  intricate 
discipline  but  also  was  utilized  as  a  quality  reading  resource  for  professionals  in  the 
contracting  career  field.  The  research  is  thorough  and  concise,  and  it  answers  the  primary 
research  question  through  the  coupling  of  military  doctrine  and  academic  theory. 

The  professional  report  of  Arzu  et  al.  (2010)  addresses  a  dire  need  inside  the  ACC 
for  contracting  officers  that  are  proficient  in  their  assigned  role  to  support  the  Army’s 
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activities  at  home  and  abroad  through  procurement  and  services  in  a  broad  spectrum  of 
contracts.  It  is  also  the  basis  for  this  professional  report,  which  focuses  on  a  similar  need 
in  the  USACE  for  its  contracting  officers  who  must  meet  all  the  ACC  proficiencies 
expected  of  an  Army  contracting  officer,  as  well  as  additional  knowledge  in  construction 
contracting  and  in  the  USACE’s  unique  organizational  structure  and  responsibilities. 

This  research  concentrates  on  the  academic  study  of  principles  and  applications  of 
item  writing  and  test  construction  in  order  to  create  a  practical  and  useful  PAT  to 
evaluate  USACE  contracting  officers  based  in  the  skills  identified  in  the  USACE’s 
51C/1102  Proficiency  Guide  (EP  715-1-8;  USACE,  2011a).  Based  on  the  findings  of  the 
Arzu  et  al.  research  team,  the  research  team  of  this  report  conducted  a  similar 
examination  of  topical  literature  to  verify  their  findings,  as  well  as  to  explore  the 
possibility  of  new  resources  published  in  the  last  two  years.  This  research  team’s  review 
indicated  that  the  theories  and  studies  the  previous  research  team  used  are  still  the  most 
contemporary  resources  available.  This  enabled  the  research  team  of  this  report  to 
proceed  in  developing  a  PAT  in  a  manner  parallel  to  the  one  for  the  ACC. 

The  USACE  contracting  subject  matter  is  extensive  in  both  its  historical  timeline 
and  breadth  of  subject  matter.  In  this  literature  review,  the  researchers  primarily  address 
contemporary  issues  facing  the  ACC,  as  well  as  US  ACE- specific  topics  in  the  21st 
century.  The  review  probes  the  problems  and  challenges  of  contracting  with  a  focus  on 
construction,  Architect-Engineer,  and  contingency  contracting.  The  matters  of  training 
and  preparation  are  reviewed  through  the  lens  of  contemporary  business  strategy, 
“institutional  Army”  methods,  Defense  Acquisition  University  (DAU)  courses,  and  the 
newly  developed  soldier’s  manual  of  common  tasks  (EP  715-1-8;  USACE,  2011)  to 
identify  strengths  to  build  upon  and/or  deficiencies  to  overcome  in  the  development  and 
preparation  of  USACE  contracting  officers. 

B.  VALUE  WEB  THEORY 

How  does  the  ACC  improve  its  core  competency  in  defense  contracting?  And 
how  does  it  assess  the  proficiency  of  its  employees  to  accomplish  the  mission?  An 
effective  means  of  measuring  the  performance  of  Army  contracting  is  to  utilize  business 
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models  addressing  facets  of  efficiency,  structure,  and  value.  From  the  evaluation  we  can 
justify  what  factors  require  attention  and  what  scope  of  attention  is  necessary. 

Brandenburger  and  Nalebuff  (2000)  discussed  the  Value  Web  theory  for 
effectively  analyzing  the  efficiency  and  strategy  of  a  firm  in  order  to  develop  a  model  of 
future  operations.  Scrutinizing  the  ACC  under  the  Value  Web  model  reveals  the 
interdependence  between  all  parties  involved  and  how  change  is  an  expected  factor  in  the 
theory  (Figure  1).  The  model  shows  how  a  participant  in  the  Value  Web  can  hold 
multiple  roles  simultaneously.  “A  key  insight  of  this  theory  is  the  importance  of  focusing 
on  others — namely,  allocentrism.  To  look  forward  and  reason  backward,  you  have  to  put 
yourself  in  the  shoes  of  other  players”  (Brandenburger  &  Nalebuff,  2000,  p.  58).  The  key 
takeaway  is  to  know  what  attributes  or  contributions  each  member  in  the  net  [web]  brings 
to  the  others  (Brandenburger  &  Nalebuff,  2000).  The  concept  of  interdependencies 
between  roles  in  the  model  builds  a  framework  of  rules  contracting  officers  should 
observe  during  business  interactions  to  build  creative  solutions  to  predicaments, 
questions,  or  new  developments  encountered. 

According  to  game  theory,  there  are  five  [elements]:  players,  added 
values,  rules,  tactics,  and  scope — P.A.R.T.S.  for  short.  These  five 
elements  fully  describe  all  interaction,  both  freewheeling  and  rule-based. 

To  change  the  game,  you  have  to  change  one  or  more  of  these  elements. 

Players  come  first.  As  we  saw  in  the  Value  Net,  the  players  are 
customers,  suppliers,  substitutors,  and  complementers.  None  of  the  players 
are  fixed.  Sometimes  it’s  smart  to  change  who  is  playing  the  game.  That 
includes  yourself. 

Added  values  are  what  each  player  brings  to  the  game.  There  are 
ways  to  make  yourself  a  more  valuable  player — in  other  words,  to  raise 
your  added  value.  And  there  are  ways  to  lower  the  added  values  of  other 
players. 

Rules  give  structure  to  the  game.  In  business,  there  is  no  universal 
set  of  rules;  a  rule  might  arise  from  law,  custom,  practicality,  or  contracts. 

In  addition  to  using  existing  rules  to  their  advantage,  players  may  be  able 
to  revise  them  or  come  up  with  new  ones. 
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Tactics  are  moves  used  to  shape  the  way  players  perceive  the  game 
and  hence  how  they  play.  Sometimes,  tactics  are  designed  to  reduce 
misperceptions;  at  other  times,  they  are  designed  to  create  or  maintain 
uncertainty. 

Scope  describes  the  boundaries  of  the  game.  It’s  possible  for 
players  to  expand  or  shrink  those  boundaries. 

Successful  business  strategies  begin  by  assessing  and  then 
changing  one  or  more  of  these  elements.  PARTS  does  more  than  exhort 
you  to  think  out  of  the  box.  It  provides  the  tools  to  enable  you  to  do  so. 
(Brandenburger  &  Nalebuff,  2000,  p.  61) 


P.A.R.T.S.:  Allocentricviewof  game 
strategy 

PLAYERS:  Entities  participating  in 
the  Value  Net 

ADDED  VALUE:  What  players  bring 
to  game 

RULES:  Give  structure  to  game 
TACTICS:  Shape  the  way  players 
perceive  game 

SCOPE:  Describe  boundaries  of 
game 


PLAYER:  Maintaining  the 
monopoly  of  service  eliminates 
competition  against  firm 


/ 


CUSTOMERS 

US  Army  -  FEDGOV 
Taxpayers 


RULES:  FEDGOV  implements  laws 
and  regulations  controlling  business 
for  the  better  of  the  taxpayer; 

RULES:  FEDGOV  demands  win-lose 
with  enemy,  but  can  negotiate  a  win- 
win  peace  talks  if  enemy  agrees 
TACTICS:  Decides  how  game  will  be 
played 


SUBSTITUTORS 

Monopoly 

Outsourcing  option? 


ADDED  VALUE:  Reduced  ft  of  projects 
drives  down  price  through 
competition,  lowering  added  value  of 
contractors; 

ADDED  VALUE:  Changing  contract 
types  or  length  of  service  could 
further  reduces  contractor  fees 


FIRM 

Contracting  Command 


COMPLEMENTORS 

Enemy 

Contractors 


PLAYER:  Contractors  provide 
competition,  reducing  prices; 
PLAYER:  Enemy  affects  what 
and  how  much  is  needed  and 
produced 

RULES:  Enemy  affects 
industry  changes 
RULES:  Enemy  could 
negotiate  with  FEDGOV  for 
win-win  peace  talks,  reducing 
costs  to  Army  &  FEDGOV 
SCOPE:  Change  in  enemy 
fight  =  change  in  Army  needs 


Figure  1.  Adapted  from  the  Value  Web  Model 

(After  Brandeburger  &  Nalebuff,  2000,  p.  60) 
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In  the  context  of  this  model,  Hamel  and  Prahalad  (1994)  gave  an  example:  “On 
any  given  day,  AT&T  might  find  Motorola  to  be  a  supplier,  a  buyer,  a  competitor,  and  a 
partner”  (Brandeburger  &  Nalebuff,  2000,  p.  60).  Hamel  and  Prahalad  (1994)  explained 
the  possible  relationships  between  participants  and  how  advantage  or  leverage  can  be 
produced.  Explaining  the  fluidity  of  roles  gives  insight  for  leaders  mentoring  contracting 
officers  in  their  wide  range  of  responsibilities.  This  model  encourages  thinking  outside 
the  entrenched  mindset  (the  typical  “us  versus  them”  or  “friends  and  enemies” 
perspective  of  relationships),  a  mindset  that  will  not  function  effectively  on  the  rapidly 
changing  battlefield. 

Contractors  hold  two  positions  in  this  model.  This  duality  of  roles  reinforces  the 
fact  that  contract  management  by  suppliers  and  proposal  competition  (complementors) 
are  key  products  of  successful  operation/interdependence  of  the  firm.  This  model  discards 
the  “us  versus  them”  roles  to  reveal  a  mutual  exchange  affecting  both  parties.  Iniquity  by 
either  participant  results  in  reduced  performance  of  the  system  as  a  whole,  which 
negatively  affects  the  Agency  (or  federal  govemment/taxpayers). 

In  the  case  of  contractor/contracting  officer,  improving  the  competencies  of  a 
participant  is  mutually  beneficial  and  improves  the  firm’s  operations.  This  theory  implies 
that  competency  at  a  role  is  not  enough,  but  an  aptitude  for  discovering  mutually 
beneficial  exchanges  is  necessary  to  produce  a  consistently  evolving  and  improving  firm. 
The  evolution  of  the  firm  dictates  to  what  extent  success  is  achieved  throughout  the 
model.  This  evolution  will  initially  come  in  the  form  of  training  and  mentorship  of 
contracting  officers  to  achieve  increased  knowledge  of  and  familiarity  with  the 
operational  opportunities  best  suited  for  the  improvement  of  participants  in  the  Value 
Web. 


1.  Strategy  and  Competitive  Advantage 

According  to  Ghemawat  and  Rivkin  (1998),  competitive  advantage  is  defined  as 
the  wedge  between  a  buyer’s  willingness  to  pay  and  the  costs  a  firm  incurs.  Additionally, 
they  define  added  value  as  “the  maximal  value  created  by  all  participants  in  a  transaction 
minus  the  maximal  value  that  could  be  created  without  the  firm”  (Ghemawat  &  Rivkin, 
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1998,  p.  6).  Strategically  valuable  resources  are  what  gives  a  firm  a  competitive  edge  and 
are  what  enable  a  firm  to  perform  activities  better  or  more  cheaply  than  rivals.  These 
resources  can  be  physical  assets  (a  prime  location),  intangible  assets  (a  strong  brand),  or 
capabilities  (a  superefficient  manufacturing  process)  (Collis  &  Montgomery,  2008, 
p.  142).  Although  a  contracting  command  is  not  a  for-profit  business,  it  must  still 
maintain  a  competitive  advantage.  The  strategically  valuable  resources  of  a  contracting 
firm  are  the  competencies  of  individuals.  “Firms  that  generate  competitive  advantages 
typically  do  so  by  devising  strategies  that  neutralize  the  unattractive  features  of  their 
industry  and  exploit  the  attractive  features”  (Ghemawat  &  Rivkin,  1998,  p.  4). 

Strategically  valuable  resources  have  five  characteristics:  (1)  they  are  difficult  to 
copy;  (2)  they  depreciate  slowly;  (3)  the  company — not  employees,  suppliers,  or 
customers — controls  their  value;  (4)  they  cannot  be  easily  substituted;  and  (5)  they  are 
superior  to  similar  resources  that  competitors  own  (Collis  &  Montgomery,  2008,  p.  142). 
As  a  contracting  command,  the  USACE  provides  a  valuable  contracting  service  to  the 
Army  through  a  small  community  of  professionals.  This  service  requires  several  years  of 
schooling  and  training/experience  to  produce  an  efficient  and  capable  contracting  officer. 
The  USACE  organizes/structures  these  professionals  to  maximize  accountability  and 
professional  development  of  its  members.  Figure  2  demonstrates  the  USACE  path  to  the 
sustained  competitive  advantage  necessary  to  optimize  performance  of  the  firm. 
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Figure  2.  Sustained  Competitive  Advantage  Model 
(After  Rothaermel,  2013,  p.  91) 
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2.  Application 

As  Arzu  et  al.  (2010)  stated,  the  development  of  a  PAT  addresses  critical  aspects 
of  the  key  failures  described  in  the  Gansler  et  al.  (2007)  report.  Researching  and 
identifying  the  important  skills  and  tasks  required  to  improve  personnel  performance 
enabled  their  team  to  create  an  assessment  tool  that  was  both  timely  and  pertinent  to  the 
needs  of  the  contracting  command.  Referring  to  the  research  by  Tigges  and  Snyder 
(1993),  Lasch  (2002),  and  Kirstein  (2003),  they  identified  tasks  necessary  to  adequately 
accomplish  the  responsibilities  of  a  contingency  contracting  officer  (CCO).  Using 
military  doctrine  as  their  basis  for  product  architecture,  Arzu  et  al.  (2010)  integrated  test 
construction  standards  and  effective  learning  principles  into  the  format  of  their  PAT. 
Further  detail  on  this  effective  template  will  be  discussed  in  the  Testing  and  Evaluation 
section  (Chapter  II,  Section  C)  of  this  report. 

Contracting  knowledge  areas  within  the  USACE  are  common  to  all  Army  51Cs, 
but  with  additional  topics  including  construction,  Architect-Engineer  contracts,  and  the 
USACE  construction  format  which  is  a  variant  of  the  Uniform  Contract  Format  (UCF) 
(standardized  format  used  in  the  preparation  of  a  solicitation  and  contract).  This  enabled 
the  research  team  to  compose  a  similar  list  that  incorporates  applicable  tasks  from  the 
USACE  51C/1102  Proficiency  Guide  (EP  715-1-8;  2011a).  This  list,  along  with 
guidance  from  the  USACE  Purple  Book  (U.S.  Army  Corps  of  Engineer  Learning  Center 
[ULC],  2011),  a  course  catalog  for  the  Proponent- Sponsored  Engineer  Corps  Training 
(PROSPECT),  is  used  for  including  pertinent  subject  matter  into  the  research  team’s 
PAT. 


C.  TESTING  AND  EVALUATION 

A  comprehensive  review  of  test  and  assessment  concepts  facilitated  the  research 
group  in  determining  how  best  to  develop  and  design  an  effective  PAT  aligned  with  core 
Army  training  standards  exhibiting  relevancy  to  specific  tasks,  conditions,  standards,  and 
performance  measures  delineated  in  the  51C/1102  Proficiency  Guide  (EP  715-1-8; 
USACE,  2011a). 
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1.  Training  and  Assessment 

Arzu  et  al.  (2010)  utilized  Field  Manuals  (FM)  7-0  ( Training  the  Force;  DoA, 
2002),  7-1  ( Battle  Focused  Training;  DoA,  2003),  and  Army  Regulation  350-1  ( Army 
Training  and  Leader  Development;  DoA,  2009)  as  the  three  principal  references 
applicable  to  current  Army  doctrine  in  the  development  of  their  PAT  for  CCOs  (Arzu  et 
al.,  2010).  All  three  of  these  references  correspondingly  provide  the  primary  framework 
to  which  the  development  of  a  suitable  PAT  designed  specifically  for  construction, 
Architect-Engineer,  and  contingency  contracting  is  to  be  based.  Army  Regulation  350-1 
(DoA,  2009)  serves  as  a  collection  of  policy  and  guidance  explicitly  for  Army  training 
and  leader  development  standards.  FM  7-0  (DoA,  2002)  acts  as  the  Army’s  capstone 
training  policy  pertinent  to  all  units,  at  all  levels,  and  in  all  components.  Ultimately,  the 
goal  of  FM  7-1  (DoA,  2003)  is  to  shape  Army  training  regulations  while  placing 
emphasis  on  teaching  leaders  to  think  through  the  training  process  in  an  effort  to  create 
leaders  who  understand  how  to  think  and  apply  durable  training  principles  to  their  units 
and  organizations. 

Whereas  FM  7-0  (DoA,  2002)  introduces  the  training  cycle,  the  linkage  of  Army 
training  and  leader  development,  and  the  three  domains  where  training  occurs 
(operational,  institutional,  and  self-development),  FM  7-1  (DoA,  2003)  demarcates  those 
precisely  accountable  for  training  and  training  support.  The  Army  Training  and  Leader- 
Development  Model,  shown  in  Figure  3,  illustrates  an  amalgamation  of  the  standards  set 
forth  in  FM  7-0  (DoA,  2002),  FM  7-0  (DoA,  2003),  and  Army  Regulation  350-1  (DoA, 
2009). 
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Figure  3.  Army  Training  and  Leader  Development  Model 

(From  DoA,  2009,  p.  4) 


Army  Regulation  350-1  (DoA,  2009)  states, 

The  Army  Training  and  Leader  Development  Model  portrays  interaction 
among  three  separate  but  overlapping  domains  (operational,  institutional 
and  self-development)  that  must  be  synchronized  in  order  to  achieve  the 
goal  of  trained  Soldiers,  Army  civilians,  leaders,  and  ready  units.  Training 
is  what  the  Army  does  every  day.  Training  builds  confidence  and 
competence,  while  providing  essential  skills  and  knowledge.  Leader 
development  is  the  deliberate,  continuous,  sequential,  and  progressive 
process — grounded  in  Army  values — that  develop  Soldiers  and  Army 
civilians  into  competent  and  confident  leaders  capable  of  decisive  action, 
mission  accomplishment,  and  taking  care  of  Soldiers  and  their  Families. 

All  training  and  leader  development  actions  occur  within  the  Army 
culture — a  culture  that  embraces  values  and  ethics,  the  Warrior  Ethos, 
standards,  and  enduring  principles  and  imperatives.  (DoA,  2009,  p.  3) 

FM  7-0(DoA,  2002)  further  explicates  leader  development,  describing  this  model 
as  a  way  of: 

...developing  trained  and  ready  units  led  by  competent  and  confident 
leaders.  The  model  identifies  an  important  interaction  that  trains  soldiers 
now  and  develops  leaders  for  the  future.  Leader  Development  is  a  lifelong 
learning  process.  The  three  core  domains  that  shape  the  critical  learning 
experiences  throughout  a  soldier’s  and  leader’s  career  are  the  operational, 
institutional,  and  self-development  domains.  Together,  these  domains 
interact  using  feedback  and  assessment  from  various  sources  and  methods 
to  maximize  warfighting  readiness.  Each  domain  has  specific,  measurable 
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actions  that  must  occur  to  develop  our  leaders.  The  operational  domain 
includes  home  station  training,  combat  training  center  rotations,  joint 
training  exercises,  and  operational  deployments  that  satisfy  national 
objectives.  Each  of  these  actions  provides  foundational  experiences  for 
soldiers,  leaders,  and  unit  development.  The  institutional  domain  focuses 
on  educating  and  training  soldiers  and  leaders  on  the  key  knowledge, 
skills,  and  attributes  required  to  operate  in  any  environment.  It  includes 
individual,  unit  and  joint  schools,  and  advanced  education.  The  self¬ 
development  domain,  both  structured  and  informal,  focuses  on  taking 
those  actions  necessary  to  reduce  or  eliminate  the  gap  between  operational 
and  institutional  experiences.  Throughout  this  lifelong  learning  and 
experience  process,  there  is  formal  and  informal  assessment  and  feedback 
of  performance  to  prepare  leaders  for  their  next  level  of  responsibility. 
Assessment  is  the  method  used  to  determine  the  proficiency  and  potential 
of  leaders  against  a  known  standard.  (DoA,  2002,  p.  1-5) 

Army  Regulation  350-1  (DoA,  2009)  describes  each  training  domain  as  a 
complement  to  the  other  two,  each  serving  a  vital  role  in  the  training,  leadership 
development,  and  unit  preparation  of  every  Army  soldier  and  civilian.  The  institutional 
domain  is  inclusive  of  Army  centers  and  schools  that  deliver  preliminary  training  and 
successive  functional  and  professional  military  education.  The  operational  domain 
integrates  individual,  unit,  and  organizational  training  activities  conducted  at  home 
station,  during  major  training  events  and  while  operationally  deployed.  The  self¬ 
development  domain  identifies  the  need  for  continuous,  lifelong  learning  aimed  at 
enabling  and  encouraging  personal  and  professional  growth  that  maximizes  strengths, 
overcomes  weaknesses,  and  encourages  individuals  to  attain  their  development  goals 
(DoA,  2009). 

The  PAT  developed  in  this  project  can  be  successfully  implemented  in  an  Army 
center  or  school  as  an  established  method  of  preliminary  training  and  successive 
education.  Furthermore,  it  can  be  utilized  as  a  self-development  tool  throughout  the 
professional  lifespan  of  an  Army  soldier  or  professional  civilian.  Although  the  PAT  is  not 
intended  to  fulfill  the  operational  domain,  it  can  be  effectively  introduced  in  the 
institutional  and  operational  domains,  and,  furthermore,  can  add  value  as  an  assessment 
geared  toward  improving  the  adult  learning  process. 
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Although  adult  learning  is  defined  as  the  process  of  adults  gaining  knowledge  and 
expertise,  adult  learning  theory  takes  a  more  situational  position  on  mutual  control 
(Knowles  et  al.,  1998).  Figure  4  provides  a  four-phase  framework  representative  of  the 
conflict  arising  between  the  ideal  of  individuals  taking  control  of  their  own  decision¬ 
making  and  the  reality  of  adult  limitations  in  taking  control  of  their  own  decision-making. 


Figure  4.  Theoretical  Foundation  of  Adult  Learning 
(From  Knowles  et  al.,  1998,  p.  125) 


The  Need  phase  focuses  on  goal  attainment  through  determining  what  particular 
learning  is  required.  The  Create  phase  formulates  a  strategy  and  applies  resources  to 
achieve  the  specific  learning  objective.  The  Implement  phase  implements  the  learning 
strategy  while  utilizing  the  learning  assets.  The  Evaluate  phase  provides  an  overall 
evaluation  of  the  learning  goal  and  the  method  of  accomplishing  it  (Knowles  et  al., 
1998).  Moreover,  the  adult  learning  theory  illustrated  in  Table  1  provides  “sound  advice 
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to  Human  Resource  Development  (HRD)  at  each  phase  of  the  planning  process” 
(Knowles  et  al.,  1998,  p.  131).  Table  1  summarizes  this  comprehensive  guidance  as  it 
applies  to  HRD. 


Table  1.  Sound  Advice  of  the  Adult  Learning  Theory 
(From  Knowles  et  al.,  1998,  p.  131) 


PHASE  SOUND  PRACTICE 


•  Need 


•  Engage  learners  in  this  phase  to  gain  higher  motivation. 

•  Do  not  expect  self-reported  needs  to  be  accurate  for 
either  the  individual  or  organization. 


•  Create 


•  Engage  learners  in  this  phase  to  gain  higher  validity  in 
the  selected  learning  strategies. 


•  Implement  •  Engage  learners  in  this  phase  to  better  mediate  the  actual 

learning. 

•  Evaluate  •  Engage  learners  in  this  phase  to  gain  higher  self¬ 

reflection  and  integration  of  knowledge  and  expertise 
being  sought. 


Although  some  specialists  and  researchers  in  the  field  hold  that  the  goal  of  HRD 
is  (or  should  be)  strictly  performance  improvement  on  an  organizational  level,  others 
believe  that  encouraging  learning  or  the  capacity  to  learn  is  in  and  of  itself  a  valuable 
outcome,  and  that  sponsoring  organizations  will  logically  benefit  (Knowles  et  al.,  1998). 
In  either  context,  the  PAT  built  by  the  team  of  researchers  of  this  report  will  benefit  both 
the  Soldier  (the  individual)  and  the  Army  (the  organization)  in  the  development  of  key 
construction,  Architect-Engineer,  and  contingency  procedures  and  concepts.  Although 
the  goal  of  performance-based  HRD  is  to  ensure  that  the  HRD  process  within 
organizations  contributes  to  the  goals  of  the  organizational  system  within  which  it 
functions,  it  does  not  automatically  imply  the  use  of  authoritarian  management  tactics 
(Knowles  et  al.,  1998).  In  consideration  to  this  observation,  the  authors  remained 
sensitive  to  the  fact  that  Army  training  doctrine  considers  assessment  on  an 
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organizational  level  a  commander’s  duty  (DoA,  2002).  In  other  words,  authoritarian 
management  tactics  are  expected  within  the  structure  of  Army  training  doctrine  and  play 
a  critical  role  in  the  development  of  its  Soldiers  and  civilians. 

Figure  5  demonstrates  how  the  commander,  as  the  principal  trainer,  uses  multi¬ 
echelon  methods  to  fuse  leader,  battle  staff,  and  individual  training  requirements  into 
cooperative  training  events,  while  distinguishing  an  overlap  in  training  responsibilities 
(DoA,  2002). 


Figure  5.  Overlapping  Training  Opportunities 
(From  DoA,  2002,  p.  2-1 1) 


Most  HRD  professionals  concur  that  HRD  should  focus  on  bolstering  the 
performance  requirements  of  its  host  organization  through  the  development  of  the 
organization’s  workforce  (Knowles  et  al.,  1998).  As  the  Army  develops  its  human 
resources  (its  Soldiers  and  civilians)  and  increases  knowledge  on  a  smaller  scale,  the 
Army  as  a  whole  benefits  on  an  organizational  level. 

Assessment  is  a  continuous  process  and  is  considered  both  the  beginning  and  the 
end  of  the  training-management  cycle  (Arzu  et  al.,  2010).  Commanders  are  expected  to 
teach,  coach,  and  mentor  subordinates  throughout  this  process  (DoA,  2002).  FM  7-0 
(DoA,  2002)  describes  the  commander’s  role  as  follows: 

Competent  and  confident  leaders  build  cohesive  organizations  with  a 
strong  chain  of  command,  high  morale,  and  good  discipline.  Therefore, 
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commanders  create  leader  development  programs  that  develop  warfighter 
professionalism — skills  and  knowledge.  They  develop  their  subordinates’ 
confidence  and  empower  them  to  make  independent,  situational-based 
decisions  on  the  battlefield.  Commanders  assist  subordinates  with  a  self¬ 
development  program  and  share  experienced  insights  that  encourage 
subordinates  to  study  and  learn  their  profession.  They  train  leaders  to  plan 
training  in  detail,  prepare  for  training  thoroughly,  execute  aggressively, 
and  evaluate  short-term  training  proficiency  in  terms  of  desired  long-term 
results.  Effective  leader  development  programs  will  continuously 
influence  the  Army  as  junior  leaders  progress  to  higher  levels  of 
responsibility,  (p.  2-12) 

Commanders  are  expected  to  (1)  develop  subordinates  in  the  proficiency  of 
mission  essential  tasks;  (2)  involve  themselves  personally  in  the  planning,  preparing, 
executing,  and  assessment  of  training;  (3)  demand  the  achievement  of  training  standards; 
(4)  ensure  proper  task  and  event  discipline;  (5)  foster  a  command  climate  that  is 
conducive  to  good  training;  and  (6)  eliminate  training  distractions  (DoA,  2002). 
Furthermore,  a  commander’s/leader’s  guidance  serves  as  one  of  the  four  major  steps  of 
the  Army’s  Training  Planning  Process  (depicted  in  Figure  6). 


Figure  6.  Army  Training  Planning  Process  (Training  Assessment) 

(From  DoA,  2003,  p.  4-4) 


Central  to  the  development  of  a  training  plan  is  the  Mission  Essential  Task  Fist 

(METE).  A  mission  essential  task  is  a  collective  task  in  which  an  organization  has  to  be 

proficient  to  accomplish  an  appropriate  portion  of  its  wartime  operational  mission  (DoA, 

2002).  The  METE  consolidates  these  tasks  and  sets  the  groundwork  for  the  unit’s  training 

program.  Developed  by  all  company-level  and  above  units,  and  approved  by  the 

26 


designated  wartime  commander,  the  METL  provides  an  insightful  synthesis  of  individual 
goals  and  organizational  requirements  (Do A,  2002). 

Commanders  diligently  appraise  their  unit’s  ability  to  execute  mission  essential 
tasks  based  on  personal  observations,  training  evaluation  results,  and  input  from 
subordinates;  for  battle  tasks  that  support  their  specific  METL,  commanders  then  collect 
performance  assessments  from  responsible  subordinates,  key  staff  members,  and  NCO 
leaders  (DoA,  2003).  Figure  7  illustrates  the  METL  development  process. 


Figure  7.  METL  Development  Process 
(From  DoA,  2002,  p.  3-3) 


FM  7-0  (DoA,  2002)  describes  the  many  advantages  of  applying  the  METL 
process  to  the  overall  Army  training  development: 

METL  development  is  the  catalyst  that  keeps  Army  training  focused  on 
wartime  operational  missions.  Applying  the  METL  development — 

•  Focuses  the  unit’s  training  on  essential  tasks. 

•  Provides  a  forum  for  professional  discussion  and  leader  development 
among  senior,  subordinate  and  adjacent  (peer)  commanders  concerning 
the  linkage  between  mission  and  training. 
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•  Enables  subordinate  commanders  and  key  NCOs  to  crosswalk  collective, 
leader  and  individual  tasks  to  the  mission. 

•  Leads  to  “buy-in”  and  commitment  of  unit  leaders  to  the  organization’s 
training  plan  (p.  3-2). 


All  learning  goals  in  the  USACE’s  51  C/1 102  Proficiency  Guide  (2011a)  are 
based  on  specifically  stated  conditions  and  standards,  and  further  identified  and  labeled 
as  individual  tasks.  Using  the  51C/1102  Proficiency  Guide  was  crucial  in  building  the 
physical  PAT,  as  it  provided  an  easy  and  appropriate  format  for  commanders  to  assess 
individual  performance  against  overall  mission-essential  objectives.  The  performance  of 
each  task,  from  Task  1-1  through  Task  4-6,  is  measured  on  a  “GO”  or  “NO  GO”  basis, 
which  aligns  with  rating  methods  stated  in  the  FM  7-1  (DoA,  2003). 

The  following  excerpt  from  FM  7-1  (DoA,  2003)  rates  individual  and  mission 
essential  tasks: 

ASSESSMENT  RATINGS 

The  commander’s  training  assessment  is  required  for  each  METL  task. 

Task  proficiency  is  rated  as — 

“T”  (trained):  The  unit  is  trained  and  has  demonstrated  proficiency  in 
accomplishing  the  task  to  the  Army  standard.  The  leader  judges  task 
performance  to  be  free  of  significant  shortcomings.  Training  on  “T”  tasks 
is  designed  to  sustain  proficiency  on  that  task. 

“P”  (needs  practice):  The  unit  can  perform  the  task  with  some 
shortcomings.  Performance  has  demonstrated  that  the  unit  does  not 
achieve  the  standard  without  some  difficulty  or  has  failed  to  perform  some 
task  steps  to  standard.  The  shortcomings  are  not  severe  enough  to  require 
complete  retraining.  Only  refresher  training  is  required. 

“U”  (untrained):  The  unit  cannot  demonstrate  an  ability  to  achieve 
wartime  proficiency.  The  leader  prepares  a  comprehensive  plan  to  train  all 
supporting  tasks  not  executed  to  standard.  Unless  the  task  is  a  new  METL 
task,  a  rating  of  “U”  indicates  a  serious  training  deficiency  and  reflects  on 
the  unit’s  wartime  readiness  posture. 
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EVALUATION  RATINGS 


Evaluation  ratings  are  given  for  specific  task  proficiency  and  should  not 
be  confused  with  leader  assessments.  Evaluation  ratings  are  ratings 
assigned  directly  to  the  performance  of  a  specific  task  or  component  steps 
of  a  task.  The  standard  evaluation  ratings,  discussed  further  in  Chapter  6, 
are  as  follows: 

“GO”:  The  task  or  performance  step  of  a  task  was  performed  to  standard. 

A  rating  of  GO  is  normally  awarded  if  all  steps  in  the  task  are  passed. 

“NO  GO”:  The  task  or  any  performance  step  in  the  task  was  not 
performed  to  standard.  (DoA,  2003,  p.  4-6) 

By  utilizing  the  PAT  developed  for  this  project,  commanders  are  better  equipped 
to  cultivate  the  management  of  their  units  and  assess  performance  of  their  human 
resources,  while  they  are  also  afforded  the  ability  to  collect  valuable  feedback  geared 
toward  the  explicit  improvement  of  both  the  construction,  Architect -Engineer,  and 
contingency  contracting  community  and  the  assessment  tool  itself.  This  PAT  is  not 
envisioned  to  serve  as  an  all-inclusive  training  guide.  It  is  intended,  however,  to  offer 
commanders  an  instrument  to  assess  individual  performance  while  affording  individuals  a 
robust  asset  to  develop  their  contracting  aptitude. 

The  research  team  was  able  to  add  relevancy  and  structure  by  recognizing  how 
the  Army  develops  and  executes  training  programs,  assesses  education,  and  evaluates 
performance  on  a  macro  level.  The  PAT  fits  well  into  the  overall  scheme  illustrated  by 
the  Army  Training  Management  Cycle  (exemplified  in  Figure  8),  which  is  considered  to 
be  the  cornerstone  of  the  training  process  but  offers  a  strong  foundation  in  establishing 
individual  self-development. 
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Figure  8.  Army  Training  Management  Cycle 
(From  DoA,  2002,  p.  2-15) 


In  The  Adult  Learner,  Knowles  et  al.  (1998)  stated. 

If  HRD  is  to  be  aligned  with  goals  and  strategies  of  the  organization,  and 
performance  is  the  primary  means  by  which  the  goals  and  strategies  or 
organizations  are  realized,  then  it  follows  that  HRD  should  be  first  and 
foremost  concerned  with  maintaining  and/or  improving  performance  at  the 
organizational,  process,  and  individual  levels.  If  HRD  is  to  be  a  value- 
added  activity  of  the  firm  (instead  of  a  line  item  of  cost  that  is  to  be 
controlled  and  minimized),  then  HRD  practitioners  must  be  concerned 
about  performance  and  how  it  enables  organizations  to  achieve  their  goals. 

(P-  H7) 

The  Army  Training  and  Leader  Development  Strategy  (AT&LDS)  closely 
parallels  The  Adult  Learner's  vision  of  HRD  and  how  it  interconnects  with  organizational 
strategy  and  builds  upon  individual  performance.  Army  Regulation  350-1  (DoA,  2009, 
p.  3)  lists  the  10  goals  of  the  Army  Training  and  Leader  Development  Strategy  as 
follows: 

•  train  units  for  full  spectrum  operations; 
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•  develop  adaptive  and  competent  leaders; 

•  enable  adaptation  of  training  and  leader  development; 

•  train  and  sustain  Soldier  and  Army  civilian  skills; 

•  sustain  and  improve  effectiveness  of  Combat  Training  Centers  (CTCs); 

•  provide  training  at  home  station  and  while  deployed; 

•  provide  training  support  system  live,  virtual,  constructive  (LVC) 
(including  gaming)  enablers; 

•  increase  culture  and  foreign  language  competencies; 

•  provide  supporting  and  integrating  capabilities;  and 

•  resource  the  AT&LDS. 

It  is  the  primary  goal  of  this  research  team  to  develop  a  PAT  that  will  be  gainfully 
employed  as  a  flexible  instrument  used  to  assess  task  performance,  aid  self-development, 
and  strengthen  US  ACE  contracting  initiatives.  A  pilot  PAT  was  constructed  to  gather 
initial  feedback  through  surveys  at  the  completion  of  the  test.  The  surveys  ask  for 
comments  on  test  difficulty,  clarity  and  validity,  and  ease  of  accomplishing  the  test  in  the 
given  amount  of  time.  As  data  returns  from  initial  testing  and  feedback  amasses  from 
additional  follow-on  surveys  conducted  by  the  US  ACE,  it  is  the  research  team’s  hope  that 
the  PAT  can  be  altered  and  improved  upon  in  future  years. 

The  research  team,  after  closely  looking  at  Army  training  doctrine  and  associated 
methodologies,  believed  the  role  of  andragogy  (the  methods  used  to  teach  adults)  was 
crucial  in  not  only  developing  the  PAT,  but  also  in  applying  learning  principles  to  adults 
in  the  professional  environment.  Andragogy  principles,  listed  in  Figure  9,  advance 
beyond  a  basic  respect  for  the  learner  and  distinguish  the  adult  learner  as  a  major  basis  of 
information  for  making  sound  decisions  in  terms  of  the  learning  process  (Knowles  et  al., 
1998).  Equally  important,  the  research  team  recognized  that  andragogy  is  a  classification 
of  components  that  is  not  intended  to  be  applied  totally  or  without  modification.  Central 
to  the  ideology  of  the  six  core  adult  learning  principles  is  flexibility  (Knowles  et  al., 
1998),  which  provides  a  significant  fit  within  commander  initiatives  and  Army  culture. 
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Figure  9.  Andragogy  in  Practice  (The  Adult  Learning  Process) 

(From  Knowles  et  al.,  1998,  p.  182) 

Learning  contracts  offer  a  resource  for  negotiating  an  appropriate  middle  ground 
between  the  external  needs  and  expectations  of  an  organization  and  the  learner’s  internal 
needs  and  interests;  more  importantly,  they  are  a  means  for  making  the  learning 
objectives  of  field-based  experience  clear  and  explicit  for  both  the  learner  and  the 
supervisor  (Knowles  et  al.,  1998).  Knowles  et  al.  (1998)  offered  eight  categorical  stages 
to  support  the  process  of  developing  a  concise  and  effective  learning  contract  for 
subordinates  and  supervisors: 

•  diagnose  your  learning  needs, 

•  specify  your  learning  objectives, 

•  specify  learning  resources  and  strategies, 

•  specify  evidence  of  accomplishment, 
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•  specify  how  evidence  will  be  validated, 

•  review  the  contract, 

•  carry  out  the  contract,  and 

•  evaluate  your  learning,  (pp.  212-216) 

Applied  within  the  organization  of  the  USACE,  the  PAT  would  serve  as  an 
effective  learning  contract  envisioned  to  closely  correspond  to  these  eight  stages. 
Piskurich,  Beckschi,  and  Hall  (2000)  justified  the  increased  use  of  learning  contracts  for 
the  following  reasons: 

•  to  provide  more  appropriate  learning  for  individuals  with  diverse 
backgrounds,  experiences,  and  learning  styles; 

•  to  meet  the  needs  of  learners  in  specialized  areas; 

•  to  meet  needs  of  learners  in  rapidly  changing  fields  when  no  appropriate 

curriculum  or  training  is  available; 

•  to  meet  the  needs  of  learners  at  a  distance; 

•  to  save  training  dollars;  and 

•  to  develop  self-directed,  reflective,  continuing  learners  who  can  contribute 
to  the  success  and  growth  of  the  organization,  (pp.  397-398) 

In  this  segment  of  the  literature  review,  the  research  team  scrutinized  the  close 
relationship  existent  between  Army  doctrine  and  leadership  development  theories. 
Moreover,  the  research  team  revealed  practical  significance  in  the  formation  of  learning 
contracts  and  progression  of  self-development  initiatives.  These  interactions  served  the 
research  team  in  building  a  developmental  format  for  the  PAT,  as  well  as  identified  the 
significant  importance  of  a  PAT  in  weighing  individual  performance,  refining  leadership 
approaches,  and  aiding  in  organizational  success.  Similar  to  the  format  outlined  in  the 
PAT  project  prepared  by  Arzu  et  al.  in  2010,  the  subsequent  segments  of  the  literature 
review  explore  core  test  principles,  test  construction  methods,  test  blueprints,  and  item 
writing. 
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2.  Core  Test  Principles 

The  Standards  for  Educational  and  Psychological  Research  (American 
Psychological  Association  [APA],  American  Educational  Research  Association  [AERA], 
and  National  Council  on  Measurement  in  Education  [NCME],  1974)  provides  a  definition 
of  a  test: 

A  test  is  a  special  case  of  an  assignment  procedure.  It  may  be  thought  of  as 
a  set  of  tasks  or  questions  intended  to  elicit  particular  types  of  behavior 
when  presented  under  standardized  conditions  and  to  yield  scores  that  will 
have  desirable  psychometric  properties  such  as  high  reliability  and  high 
validity,  (p.  2) 

Similarly,  in  Educational  Measurement  and  Evaluation,  Nunnally  and  Ator 
(1972)  defined  a  test  as  a  standardized  condition  that  provides  an  individual  with  a  score 
and  is  designed  and  controlled  to  serve  a  definite  purpose.  Nunnally  and  Ator  (1972) 
listed  the  most  essential  ingredients  of  standardization  as  it  applies  to  testing  as  follows: 

•  all  students  should  answer  the  same  questions; 

•  instructions  should  be  clear,  and  the  same  instructions  should  be  given  to 
all  students; 

•  no  student  should  be  given  any  advantage  not  given  to  all  students;  and 

•  a  predetermined  system  of  scoring  should  uniformly  be  applied  to  the 
answers  of  all  students. 

Standardization  is  the  chief  determinant  of  a  test’s  reliability  and  validity.  Though 
it  was  conclusive  among  the  sources  that  reliability  and  validity  were  the  two  most 
critical  test  characteristics,  several  authors  suggested  supplementary  variants.  Wood 
(1960)  listed  scoring  economy  and  adequacy  of  content  sampling.  Marshall  and  Hales 
(1972)  proposed  objectivity,  fairness,  practicality,  and  balance.  Brennan  (2006)  made 
reference  to  efficiency. 

A  test  is  considered  to  be  reliable  if  it  generates  highly  exact  indications  of 
students’  standings  with  respect  to  one  another;  if  a  test  is  not  highly  reliable,  a  zone  of 
uncertainty  must  be  deliberated  in  interpreting  particular  scores  (Nunnally  &  Ator,  1972). 
Marshall  and  Hayes  (1972)  referred  to  reliability  as  the  degree  of  consistency  among  test 
scores.  There  are  multiple  methods  that  initiate  reliability: 
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Reliability  can  be  defined  in  several  ways — such  as  the  proportion  of 
observed-score  variance  that  is  true-score  variance,  the  squared  correlation 
between  true  scores  and  observed  scores,  or  the  correlation  between 
observed  scores  on  two  parallel  tests.  Several  estimates  of  reliability  are 
widely  used,  including  test/retest,  parallel-forms,  alternate  forms,  and 
internal-consistency  estimates.  (Allen  &  Yen,  2001,  p.  91) 

In  A  Handbook  of  Test  Construction:  Introduction  to  Psychometric  Design ,  Kline 
(1986)  suggested  sources  of  unreliability  that  include  guessing,  poor  test  instructions,  or 
test  length. 

The  reliability  of  a  test  or  the  accuracy  with  which  it  measures  something  in  a 
certain  group  is  a  critical  determinant  of  a  test’s  validity  (Wood,  1960).  A  test  is 
considered  to  be  valid  if  it  serves  its  intended  function  well  (Nunnally  &  Ator,  1972).  It  is 
possible  for  a  test  to  be  reliable  but  concurrently  invalid.  Wood  (1960)  described  the 
relationship  between  reliability  and  validity: 

If  the  test  is  not  measuring  whatever  it  measures  consistently,  then  it 
cannot  be  valid  for  any  purpose.  On  the  other  hand,  it  may  measure 
something  with  a  high  degree  of  reliability  without  being  at  all  useful  for 
the  purpose  for  which  it  is  intended.  In  other  words,  a  test  may  have  even 
a  perfect  degree  of  reliability  and  no  validity  whatever  for  some  particular 
use.  A  test  of  a  given  degree  of  reliability  will  ordinarily  have  different 
validities  for  different  purposes.  Thus  the  concept  of  validity  makes  sense 
only  if  we  specify  the  purpose.  Note,  moreover,  that  the  notion  of  purpose 
here  must  include  the  group  which  is  to  take  the  test.  (p.  16) 

Validity  is  assessed  based  on  content  validity,  construct  validity,  and  criterion- 
related  validity  (Allen  &  Yen,  2001).  Content  validity  can  be  determined  based  on 
thorough  scrutiny  of  an  assessment  proportional  to  what  the  assessment  is  designed  to 
measure  and  is  absolutely  required  for  an  acceptable  assessment;  essentially,  the  validity 
of  an  assessment  depends  upon  the  acceptability  with  which  a  body  of  intellectual  content 
is  covered  (Nunnally  &  Ator,  1972).  Construct  validation  entails  intertwining  a  web  of 
meaningful  relations  between  a  new  measure  and  other  supposed  measures  of  an  identical 
attribute  (Nunnally  &  Ator,  1972).  Criterion-related  validity  is  interchangeable  with  the 
term  predictive  validity  and  correlates  the  scores  of  a  specific  criterion  (the  thing  to  be 
predicted)  and  the  actual  test  (Nunnally  &  Ator,  1972). 
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According  to  the  Standards  for  Educational  and  Psychological  Research  (APA, 
AERA,  &  NCME,  1974),  all  tests  involve  a  test  writer,  test  taker,  and  test  user.  A  test 
user  is  defined  as  one  who  chooses  tests,  interprets  scores,  or  makes  decisions  based  on 
test  scores  (APA,  AERA,  &  NCME,  1974,  p.  1).  Basic  user  qualifications  apply  when 
tests  are  specifically  used  for  decisions,  as  such  uses  typically  require  additional  technical 
qualifications;  the  legitimacy  of  a  test  is  dependent  on  the  technical  skill  and  knowledge 
essential  to  appraise  the  validity  of  various  types  of  inferences  (APA,  AERA,  &  NCME, 
1974,  p.  58).  See  Appendix  A  for  a  listing  of  essential  principles  for  test  users. 

Based  on  the  research  of  Nunnally  and  Ator  (1972),  and  Wood  (1960), 
constructing  a  PAT  that  meets  the  standards  of  educational  and  psychological  research 
required  clearly  identifying  participants  involved  in  the  PAT’s  production,  management, 
and  execution.  The  participants  are  the  research  team  (test  writer),  the  USACE  (test  user), 
and  the  USACE  51C/1102s  (test  taker).  The  PAT  would  be  assessed  by  the  USACE 
leadership  for  how  effectively  the  subject  matter  is  covered,  thereby  validating  the  PAT. 

3.  Test  Construction 

Downing  and  Haladyna’s  (2006)  12  steps  for  effective  test  development  (listed  in 
Appendix  B)  were  employed  in  the  construction  of  the  PAT.  Systematically  following 
these  steps  greatly  assisted  the  research  team  in  providing  a  structured  process  for 
creating  an  effective  testing  program  while  also  encouraging  maximum  test  validity. 
According  to  Downing  and  Haladyna  (2006), 

Effective  test  development  requires  a  systematic,  well-organized  approach 
to  ensure  sufficient  validity  evidence  to  support  the  proposed  inferences 
from  the  test  scores.  A  myriad  of  details  and  issues,  both  large  and  small, 
comprise  the  enterprise  usually  associated  with  the  terms  test  development 
and  test  construction.  All  of  these  details  must  be  well  executed  to  produce 
a  test  that  estimates  examinee  achievement  or  ability  fairly  and 
consistently  in  the  content  domain  purported  to  be  measured  by  the  test 
and  to  provide  documented  evidence  in  support  of  the  proposed  test  score 
inferences,  (p.  3) 
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In  order  to  achieve  effective  test  development,  the  research  team  employed  the  12  steps 
itemized  by  Downing  and  Haladyna  (2006,  p.  5).  The  research  team’s  approach  for  each 
of  these  items  is  discussed  in  detail  in  Chapter  IV  of  this  report. 

Arzu  et  al.  (2010),  in  building  their  CCO  PAT  in  2010,  found  that  focusing  on  test 
blueprints  and  item  writing  would  be  most  beneficial  in  the  context  of  test  construction 
and  within  the  overall  scope  of  their  project.  The  research  team  of  this  project  also 
exhibits  a  strong  focus  in  contingency  contracting,  but  adds  complexity  in  the  Architect- 
Engineer  and  construction  contracting  niche  fields.  Test  blueprints  and  item  writing  add 
equal  value  to  the  PAT  in  terms  of  test  development  and  construction. 

4.  Test  Blueprints 

The  process  of  creating  test  specifications  directs  comprehensive  test 
development  activities  and  completes  the  operational  planning  for  tests  in  an  organized 
method.  A  test  blueprint  accurately  identifies  the  percentage  of  test  questions  to  be 
allocated  to  each  major  and  minor  content  area,  and  what  percentage  of  these  questions 
will  be  designed  to  assess  specific  cognitive  knowledge  levels  (Downing  &  Haladyna, 
2006).  Both  Downing  and  Haladyna  (2006)  and  Brennan  (2006)  used  the  terms  test 
blueprints  and  specifications  interchangeably. 

Specifications,  or  blueprints,  are  responsible  for  an  assessment’s  content,  form, 
and  functional  requirements  (Brennan,  2006).  Marshall  and  Hales  (1972)  described  the 
importance  of  a  table  of  specifications  as  follows: 

Since  the  primary  purpose  of  classroom  testing  is  to  obtain  individual 

measures  for  evaluating  students  regarding  their  meeting  of  the 

instructional  objectives,  a  blueprint  for  selecting  appropriate  test  items 

should  be  developed,  (p.  10) 

The  research  team  used  the  principles  from  Downing  and  Haladyna  (2006)  to  develop  a 
test  blueprint  which  include: 

•  the  type  of  testing  format  to  be  used  (selected  response  versus  constructed 
response), 

•  the  total  number  of  test  items  as  well  as  the  type  or  format  of  test  items, 

•  the  cognitive  classification  system  to  be  used, 
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•  whether  or  not  the  test  items  or  performance  prompts  will  contain  visual 
stimuli, 

•  the  expected  item  scoring  rules, 

•  how  test  scores  will  be  interpreted  (norm  or  criterion  referenced),  and 

•  the  time  limit  for  each  item.  (p.  9) 

The  research  team  utilized  a  single-best  answer  from  multiple-choice  selections.  The 
details  of  the  research  team’s  approach  to  the  test  blueprint  are  discussed  in  Chapter  IV. 

A  teacher  must  be  able  to  classify  instructional  objectives  into  two  dimensions: 
content  and  cognitive  (Marshall  &  Hales,  1972).  Downing  and  Haladyna  (2006)  defined 
content  as: 

a  conceptual  framework  that  delineates  students’  knowledge,  constructs, 
skills,  concepts,  reasoning,  and,  in  some  cases,  disposition  that  are 
intended  to  be  measured  by  the  test.  The  need  for  clear  content 
specification  and  the  assurance  that  the  inferences  made  from  the  test 
scores  about  student  achievement  are  appropriate  for  the  stated  purposes 
of  the  test  are  both  critical  to  test  validity.  Because  of  the  close  link 
between  these  factors  in  establishing  the  validity,  content  specification  is 
integral  to  the  content  validation  process,  (p.  156) 

In  order  to  frame  the  content  of  a  test,  the  research  team  had  to  answer  the 
following  questions:  What  content  would  the  test  cover?  How  complex  to  make  the  test 
items?  What  range  of  content  is  appropriate?  How  much  emphasis  to  give  to  specific 
focus  areas?  (Downing  &  Haladyna,  2006).  The  research  team  limited  the  content  to  the 
51C/1102  Proficiency  Guide  and  listed  references  for  each  subject  area.  Emphasis  and 
complexity  of  test  items  were  focused  on  the  performance  steps  and  standards  identified 
for  each  task. 

Bloom’s  (1956)  research  efforts  explored  the  taxonomy  of  educational  objectives, 
while  concentrating  principally  on  the  cognitive  domain,  which  consists  of  knowledge, 
comprehension,  application,  analysis,  synthesis,  and  evaluation.  The  revised  version  of 
Bloom’s  taxonomy,  depicted  in  the  Figures  10  and  11,  focuses  strongly  on  the  knowledge 
dimension  useful  in  the  development  of  the  PAT  (Downing  &  Haladyna,  2006). 
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CATEGORIES 

&  COGNITIVE  ALTERNATIVE 

PROCESSES  NAMES  DEFINITIONS  AND  EXAMPLES 


1. 

REMEMBER 

Retrieve  relevant  knowledge  from  long-term  memory 

Locating  knowledge  in  long-term  memory  that  is  consistent  with 

1.1 

RECOGNITION 

Identifying 

presented  material  (e.g..  Recognize  the  dates  of  important  events 
in  U.S.  history) 

1.2 

RECALLING 

Retrieving 

Retrieving  relevant  knowledge  from  long-term  memory  (e.g..  Recall 
the  dates  of  important  events  in  U.S.  history) 

2. 

UNDERSTAND 

Construct  meaning  from  instructional  messages,  including  oral,  writen,  and  graphic 
communication 

2.1 

INTERPRETING 

Clarifying 

Paraphrasing 

Representing 

Translating 

Changing  from  one  form  of  representation  (e.g.,  numerical)  to 
another  (e.g.,  verbal)(e.g..  Paraphrase  important  speeches  and 
documents) 

2.2 

EXEMPLIFYING 

Illustrating 

Instantiating 

Finding  a  specific  example  or  illustaration  oof  a  concept  or  principle 
(e.g..  Give  examples  of  various  artistic  painting  styles) 

2.3 

CLASSIFYING 

Categorizing 

Subsuming 

Determining  that  something  belongs  to  a  category  (e.g.,  concept  or 
principle)(e.g..  Classify  observed  or  described  cases  of  mental 
disorders) 

2.4 

SUMMARIZING 

Abstracting 

Generalizing 

Abstracting  a  general  theme  or  major  point(s)(e.g..  Write  a  short 
summary  of  the  events  protrayed  on  a  videotape) 

2.5 

INFERRING 

Concluding, 

Extrapolating 

Interpolating 

Predicting 

Drawing  a  logical  conclusion  from  presented  information  (e.g.,  In 
learning  a  foreign  language,  infer  grammatical  principles  from 
examples) 

2.6 

COMPARING 

Contrasting 

Mapping 

Matching 

Detecting  correspondences  between  two  ideas,  objects,  and  the 
like  (e.g..  Compare  historical  events  to  contemporary  situations) 

2.7 

EXPLAINING 

Constructing 

models 

Constructing  a  cause-and-effect  model  of  a  system  (e.g..  Explain  the 
causes  of  important  18th-century  events  in  France) 

3. 

APPLY 

Carry  out  or  use 

a  procedure  in  a  given  situation 

3.1 

EXECUTING 

Carrying  out 

Applying  a  procedure  to  a  familiar  task  (e.g..  Divide  one  whole 
number  by  another  whole  number,  both  with  multiple  digits) 

3.2 

IMPLEMENTING 

Using 

Applying  a  procedure  to  an  unfamiliar  task  (e.g..  Use  Newton's 

Second  Law  in  situations  in  which  it  is  appropriate) 

Figure  10.  Revised  Bloom’s  Taxonomy  of  Learning  Objectives  (I) 
(Adapted  from  Anderson  et  al.,  2002,  p.  67) 
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CATEGORIES 

&  COGNITIVE  ALTERNATIVE 


PROCESSES 

NAMES 

DEFINITIONS  AND  EXAMPLES 

4. 

ANALYZE 

Break  material  into  its  constituent  parts  and  determine  how  the  parts  relate  to  one 
another  and  to  an  overall  structure  or  purpose 

4.1 

DIFFERENTIATING 

Discriminating 

Distinguishing  relevant  from  irrelevant  parts  or  important  from 

Distinguishing 

unimportant  parts  of  presented  material  (e.g.,  Distinguished 

Focusing 

between  relevant  and  irrelevant  numbers  in  a  mathematical  word 

Selecting 

problem) 

4.2 

ORGANIZING 

Finding 

Determining  how  elements  fit  orfunction  within  a  structure  (e.g.. 

Coherence 

Structure  evidence  in  a  historical  description  into  evidence  for  and 

Intergrating 

Outlining 

Parsing 

Structuring 

against  a  particular  historical  explanation) 

4.3 

ATTRIBUTING 

Deconstructing 

Determine  a  point  of  view,  bias,  values,  or  intent  underlying 
presented  material  (e.g..  Determine  the  point  of  view  of  the  author 
of  an  essay  in  terms  of  his  or  her  political  perspective) 

5. 

EVALUATE 

Make  judgements  based  on  criteria  and  standards 

5.1 

CHECKING 

Coordinating 

Detecting  inconsistencies  orfallacies  within  a  process  or  product; 

Detecting 

determining  whether  a  process  or  product  has  internal  consistency; 

Monitoring 

detecting  the  effectiveness  of  a  procedure  as  it  is  being 

Testing 

implemented  (e.g..  Determine  if  a  scientist's  conclusions  follow 
from  observed  data 

5.2 

CRITIQUING 

Judging 

Detecting  inconsistencies  between  a  product  and  external  criteria, 
determining  whether  a  product  has  external  consistency;  detecting 
the  appropriateness  of  a  procedure  for  a  given  problem  (e.g..  Judge 
which  of  two  methods  is  the  best  way  to  solve  a  given  problem) 

6. 

CREATE 

Put  elements  together  to  form  a  coherent  or  functional  whole,  reorganize  elements  into 
a  new  pattern  or  structure 

6.1 

GENERATING 

Hypothesizing 

Coming  up  with  alternative  hypotheses  based  on  criteria  (e.g., 
Generate  hypotheses  to  account  for  an  observed  phenomenon) 

6.2 

PLANNING 

Designing 

Devising  a  procedure  for  accomplishing  some  task  (e.g..  Plan  a 
research  paper  on  a  given  historical  topic) 

6.3 

PRODUCING 

Constructing 

Inventing  a  product  (e.g.,  Build  habitats  for  a  specific  purpose) 

Figure  11.  Revised  Bloom’s  Taxonomy  of  Learning  Objectives  (II) 
(Adapted  from  Anderson  et  al.,  2002,  p.  68) 
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The  research  team  modified  Bloom’s  Taxonomy  to  simplify  question 
classification  into  only  three  categories,  which  include  recall,  comprehension  and 
application.  This  modified  version  falls  closer  in  line  with  the  following  Marshall  and 
Hales  (1972)  table  of  specifications. 

Marshall  and  Hales  (1972)  offer  a  two-way  table  of  specifications  (see  Table  2) 
that  reflect  a  relationship  between  intellectual  activity  and  content  for  the  instructional 
objective. 


Table  2.  Table  of  Specifications 

(From  Marshall  &  Hales,  1972,  p.  12) 


Table  2.1 

Table  of  specifications 

Level  of  Objectives 

Number  of 

Weight 

Content 

Knowledge  Understanding  Application 

items  by 

30% 

40% 

30% 

content  area 

10% 

Identifying  notes 
by  name 

1* 

1 

1 

3 

35% 

Combining  rhythms 

to  form  measures 

3 

4 

3 

10 

35% 

Placing  pitches  on 

musical  staff 

3 

5 

3 

11 

20% 

Combing  pitches  and 
rhythms 

2 

2 

2 

6 

Number  of  items  by 
cognitive  classification 

9 

12 

9 

30 

*  Number  of  items  in  cell  =  row  weight  x  column  weight  x  number  of  test  items  (for  Area  I, 
Knowledge)  =  0.30 x  0.10 x  30  =  0.9  items. 


The  content  and  purpose  of  each  PAT  item  is  verified  through  classification  (by 
type  and  category)  to  accomplish  the  necessary  breadth  and  depth  of  testing.  Classifying 
the  test  items  also  ensures  that  the  PAT  has  a  satisfactory  distribution  of  item  types 
within  the  test.  This  research  provides  USACE  with  an  assessment  tool  that  efficiently 
accomplishes  the  fundamental  expectations  of  Army  training  doctrine  while  applying 
current  educational  theory. 
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5.  Item  Writing 

Although  Chapter  IV  (Methodology)  explicitly  describes  the  guidelines  the 
research  team  employed  in  writing  items  within  the  PAT,  this  section  provides  general 
background  on  item  writing.  The  basic  unit  of  a  test  is  an  item,  or  the  individual  “thing” 
that  is  scored  (Nunnally  &  Ator,  1972).  Both  Nunnally  and  Ator  (1972)  and  Wood  (1960) 
listed  the  various  identities  of  test  items  as  follows:  true  or  false,  multiple  choice, 
identification,  short-answer  essay,  problems,  matching,  rank  order,  and  completion 
(Nunnally  &  Ator,  1972;  Wood,  1960).  Many  different  methods  and  approaches  were 
recommended  throughout  the  references  based  on  the  identity  of  the  test  item  itself. 
Because  the  PAT  developed  for  this  project  uses  primarily  multiple-choice  questions,  the 
majority  of  item-writing  strategies  employed  were  either  specifically  geared  toward  the 
multiple-choice  item  or  suggested  as  a  generic  method  for  all  test  items. 

Good  items  appear  to  be  deceivingly  simple  to  the  novice  writer;  although  it  may 
be  easy  to  construct  an  item,  it  is  an  entirely  different  matter  to  construct  a  good  item 
(Marshall  &  Hales,  1972).  Osterlind  (1998)  incorporated  this  concept,  and  distinguished 
between  professional  test-item  writers  and  casual  test-item  writers.  In  an  effort  to  guide 
test-item  writers  of  various  skill  levels,  Marshall  and  Hales  (1972)  stressed  seven  core 
procedures  they  believed  played  a  critical  role  in  good  test-item  development: 

•  Allow  adequate  time  for  the  construction  of  items. 

•  An  item  should  contain  a  problem  to  be  solved. 

•  The  problem  should  be  defined  explicitly. 

•  The  problem  should  be  limited. 

•  The  directions  for  the  test  should  be  stated  explicitly. 

•  Do  not  use  optional  questions. 

•  Construct  a  detailed  key  for  each  question,  (pp.  31-32) 
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Downing  and  Haladyna  (2006)  also  produced  eight  item-development  strategies: 


•  Use  novel  material  to  test  higher  level  concepts.  Paraphrase  textbook 
language  or  language  used  during  instruction  to  avoid  testing  for  simple 
recall. 

•  Avoid  overly  specific  and  overly  general  content. 

•  Minimize  the  amount  of  reading  in  each  item. 

•  Avoid  window  dressing  (excess  verbiage). 

•  Word  the  stem  positively,  avoid  negatives  such  as  not  or  except.  If 
negatives  are  used,  use  the  word  cautiously  and  always  ensure  that  the 
word  appears  capitalized  and  boldface. 

•  Develop  as  many  effective  choices  as  you  can. 

•  Make  sure  that  only  one  of  these  choices  is  the  right  answer. 

•  Place  choices  in  logical  or  numeric  order,  (pp.  95-102) 


The  research  team  found  sufficient  guidance  for  writing  good  items.  Kline  (1986) 
identified  several  rules  for  writing  good  multiple-choice  items.  These  rules  include  using 
simple  writing;  ensuring  that  the  distractors  chosen  are  capable  of  distracting  examinees; 
having  only  one  correct  option;  avoiding  the  use  of  answers  that  clue  the  examinee  to 
other  items;  and,  finally,  testing  relevant  information  and  avoiding  trivial  information. 

Nunnally  and  Ator  (1972)  described  the  importance  of  test  items  as  they  relate  to 
evaluation: 

The  inability  to  compose  good  items  is  the  major  reason  why  some 
teachers  do  a  poor  job  of  evaluating  the  progress  of  students.  One  cardinal 
fault  of  many  sets  of  test  items  is  that  they  are  not  broadly  representative 
of  the  important  content  in  a  particular  unit  of  instruction.  Either  they  are 
overly  slanted  toward  one  or  another  aspect  of  the  content,  or  even  if  they 
are  broadly  representative,  they  tap  only  trivial  information,  e.g.,  memory 
for  miscellaneous  facts,  (p.  153) 

The  research  team  is  prioritizing  the  performance  steps  and  standards  identified  for  a 
particular  task;  therefore,  the  team  must  be  conscience  of  not  becoming  too  narrow 
minded  and  still  develop  sufficient  questions  in  related  reference  areas  in  order  to  gain 
the  broad  representation  discussed  by  Nunnally  and  Ator  (1972). 
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The  research  by  Nunnally  and  Ator  (1972),  Kline  (1986),  Downing  and  Haladyna 
(2006),  and  Wood  (1960)  provided  the  academic  understanding  for  producing  test  items 
written  in  the  most  effective  manner.  Utilizing  principles  discussed  in  the  research 
ensures  that  the  PAT  items  are  valid  and  functional,  and  that  they  clearly  convey  the 
intent  of  the  test. 

D.  CONTRACTING  DEFICIENCIES 

Based  on  the  failures  identified  in  the  Gansler  (2007)  report,  the  research  team 
designated  this  document  as  a  starting  point  in  the  literature  review.  Utilizing  sources  in 
the  Contingency  Contracting  professional  report  (Arzu  et  al.,  2010),  the  research  team 
reviewed  databases  containing  studies  or  reports  regarding  construction  contracting.  The 
USACE  Office  of  Inspector  General’s  database  provided  reports  applicable  to  the 
research  while  also  giving  the  explicit  details  of  common  challenges  in  planning, 
awarding,  and  administering  contracts.  The  DTIC  contributed  additional  sources  on 
topics  relevant  to  construction  contracting  in  a  contingency  environment.  Supplementary 
data  for  this  literature  review  were  acquired  through  reports  from  the  GAO,  USACE 
archives,  and  DoD  guidance. 

For  the  purpose  of  this  literature  review,  the  research  team  addresses  several 
topics  discussing  the  ACC,  the  major  command  primarily  responsible  to  coordinate  and 
enforce  the  training  needs  of  the  Army’s  contracting  officers.  Duties  of  a  contracting 
officer  in  the  USACE  are  in  addition  to  the  standards  expected  by  the  ACC.  The  focus  of 
ACC  training  is  to  produce  competent  and  flexible  professionals  capable  of  creating 
quality  contracts  that  ensure  the  best  possible  price  with  minimal  disturbance  to  project 
execution.  In  that  context,  the  research  team  also  reviews  in  business  strategy  terms  how 
the  ACC  (as  well  as  the  USACE)  is  in  an  unfavorable  market,  undertaking  a  low-cost 
strategy  utilizing  professional  contracting  officers  (expected  to  have  increased  training 
and  preparation)  to  accomplish  more  effective  contracts  for  its  customers. 

A  review  of  the  USACE  51C/1102  Proficiency  Guide  (USACE,  2011a)  shows  that 
contracting  officers  within  the  USACE  are  challenged  with  a  broad  spectrum  of 
competencies  in  planning,  structuring,  and  managing  contracts  of  differing  scope  and 
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types  that  cover  an  abundance  of  supplies  and  services.  In  addition  to  this  challenge, 
policies,  regulations,  thresholds,  and  standards  are  constantly  evolving  to  address 
contemporary  threats.  The  complexities  of  this  market -battlefield  are  evident  by  the 
frequency  and  magnitude  of  failures  that  occur  (Gansler,  2007;  GAO,  2012).  The 
exorbitant  costs  in  time,  money,  and  business  relations  demand  an  effective  and 
immediate  action  that  will  address  the  shortcomings  that  cause  the  costly  failures. 

1.  Complexities  Leading  to  Failure  in  Contracting 

Arzu  et  al.  (2010)  explored  databases,  professional  reports/surveys,  historical 
documents,  and  federal  manuals  and  regulations  during  their  fact-finding  search  of  needs 
inside  the  acquisition  community  (pp.  16-30).  A  recurring  theme  across  much  of  the 
reading  was  the  lack  of  education/development  in  core  competencies.  Complications 
arising  from  allegations  of  waste,  fraud,  and  abuse  and  significant  failures  in 
expeditionary  contracting  compelled  the  government  to  investigate  the  circumstances  and 
report  findings. 

In  2007,  the  report  by  the  Gansler  Commission  titled  Urgent  Reform  Required: 
Army  Expeditionary  Contracting,  the  Commission  on  Army  Acquisition  and  Program 
Management  in  Expeditionary  Operations  (led  by  Dr.  Jacques  S.  Gansler,  former  Under 
Secretary  of  Defense  [Acquisition,  Technology  &  Logistics  (AT&L)]),  exposed  failures 
occurring  in  expeditionary  contracting  operations.  In  order  to  rectify  difficulties  in  the 
acquisition  process,  the  Gansler  Commission  (2007)  recommended  “four  overarching 
changes  for  the  Army”: 

•  increase  the  stature,  quantity,  and  career  development  of  military  and 
civilian  contracting  personnel  (especially  for  expeditionary  operations); 

•  restructure  the  organization  [Army]  and  restore  responsibility  to  facilitate 
contracting  and  contract  management  in  expeditionary  and  CONUS 
[Continental  United  States-based]  operations; 

•  provide  training  and  tools  for  overall  contracting  activities  in 
expeditionary  operations;  and 

•  obtain  legislative,  regulatory,  and  policy  assistance  to  enable  contracting 
effectiveness  in  expeditionary  operations,  (p.  5) 
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Additionally,  in  2007,  the  Comptroller  General  of  the  United  States  stated  in 
GAO  Report  07-1098T,  Federal  Acquisition  and  Contracting:  Systemic  Challenges  Need 
Attention ,  four  key  acquisition  challenges  affecting  agencies  in  the  U.S.  government.  The 
challenges  were  (1)  separating  wants  from  needs,  (2)  establishing  and  supporting  realistic 
program  requirements,  (3)  using  contractors  in  appropriate  circumstances  and  contracts 
as  a  management  tool,  and  (4)  creating  a  capable  workforce  and  holding  it  accountable 
(GAO,  2007;  Arzu  et  al.,  2010). 

The  Commission  on  Wartime  Contracting  (CWC),  a  congressional  commission 
tasked  to  investigate  wartime  fraud,  waste,  and  abuse  allegations  in  Iraq  and  Afghanistan 
theaters  of  operation,  cited  numerous  deficiencies  in  training  of  contracting  personnel 
(CWC,  2009).  The  CWC  report,  titled  At  What  Cost?  Contingency  Contracting  in  Iraq 
and  Afghanistan ,  recognized  training  problems  of  the  federal  civilian  and  military 
contracting  workforce.  Deficiencies  stemmed  primarily  from  manpower  and  lack  of 
sufficient  training  for  contracting  personnel  (CWC,  2009,  p.  26). 

In  a  2010  report  by  the  Department  of  Defense  Office  of  Inspector  General 
(DoDIG),  similar  concerns  are  reported  based  on  19  investigations  of  fraud  from  October 
2007  through  April  2010.  The  findings  identified  10  systemic  issues  related  to 
contracting  deficiencies  with  the  top  five  issue  areas  being  (1)  requirements,  (2)  contract 
pricing,  (3)  oversight  and  surveillance,  (4)  property  accountability,  and  (5)  financial 
management  (DoDIG,  2010).  The  deficiencies  reflect  the  recurring  failures  in 
fundamental  practices  across  the  contracting  community. 

The  2009  DoDIG  report  Summary  of  DoD  Office  of  Inspector  General  Audits  of 

Acquisition  and  Contract  Administration  summarized  major  contracting  issues  found  in 

DoDIG  reports  from  fiscal  year  (FY)  2003  to  FY  2008.  In  this  report,  researchers 

concluded  that  the  areas  of  management  oversight,  control,  and  policy  enforcement  must 

continue  to  be  strongly  emphasized  to  protect  valuable  taxpayer  funds  (DoDIG,  2009). 

The  researchers  of  the  report  also  stated  that  DoD  has  taken  numerous  actions  to  address 

identified  deficiencies  in  the  areas  of  contingency  operations  and  continuing  DoD 

problems  (DoDIG,  2009).  In  particular,  the  DoD  initiatives  addressing  contingency 

operations  included  revising  the  DAU  contingency  contracting  curriculum  as  well  as 
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fielding  a  handbook,  Contingency  Contracting:  A  Joint  Handbook,  (DPAP,  2008)in  2008 
that  consolidated  important  contracting  information,  tools,  templates  as  well  as  training 
that  a  contracting  officer  could  quickly  reference  (DoDIG,  2009).  The  intent  of  fielding 
the  handbook  was  for  training  and  reference  while  at  home  station  or  deployed  (DoDIG, 
2009).  In  a  similar  way,  the  USACE  51  C/1 102  Proficiency  Guide  (2011a)  discussed  in 
the  following  section  is  meant  for  the  same  purposes  with  a  heavier  application  of 
construction  and  architect-engineer  contracting. 

The  DoDIG  also  looked  specifically  at  the  USACE  in  recommending  that 
contracting  oversight  of  military  construction  projects  at  Bagram,  Afghanistan,  be 
improved  (DoDIG,  2012).  The  DoDIG  report  indicated  that  contractor  performance  on 
USACE  contracts  were  not  properly  monitored  and  relied  heavily  on  the  Logistics  Civil 
Augmentation  Program  for  identification  of  deficiencies  with  construction  (DoDIG, 
2012).  Recommendations  the  DoDIG  made  for  improving  oversight  included  verification 
of  quality  control  plans  and  training  of  quality  assurance  personnel  in  a  contingency 
environment  (DoDIG,  2012). 

Contract  management  within  the  DoD  has  remained  on  the  GAO  high-risk  list 
since  1993  (Seifert  &  Ermoshkin,  2010).  The  2011  GAO’s  high-risk  area 
recommendations  for  DoD  contract  management  included  ensuring  adequate  training  of 
the  acquisition  workforce  to  meet  organizational  needs  (GAO,  2011).  The  USACE 
51C/1102  Proficiency  Guide  (2011a)  and  the  research  team’s  developed  PAT  are 
consistent  with  this  recommendation  as  tools  to  further  training  needed  within  the 
USACE. 

The  findings  of  the  Gansler  (2007)  report,  the  Arzu  et  al.  (2010)  professional 
report,  DoDIG  reports  (2009,  2010),  and  GAO  report  (2011)  describe  the  current 
operating  environment  in  contracting  as  well  as  relations  between  participants.  This 
knowledge  base  serves  as  the  input  for  modeling  how  the  firm  (this  is  a  business  term  the 
research  team  uses  in  this  report  to  refer  to  the  contracting  command)  operates.  The 
research  team  identified  model  elements  that  can  be  changed  to  improve  the  performance 
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and/or  efficiency  of  participants  and  the  firm  as  a  whole.  Some  of  the  findings  the 
research  team  presents  exceed  the  scope  of  this  professional  report  but  would  be  excellent 
topics  for  follow-on  research. 

2.  USACE  51C/1102  Proficiency  Guide 

In  2011,  the  51C/1102  Proficiency  Guide  (EP  715-1-8)  was  created  to  assist  in 
the  fundamental  development  of  USACE  members  holding  MOS  5 1C  or  1102.  It 
structures  the  essential  tasks  for  training  into  four  subject  areas:  (1)  USACE,  (2) 
Contracting  Pre-Award  Tasks  for  Construction  Contracting,  (3)  Contracting  Post-Award 
Tasks  for  Construction  Contracts,  and  (4)  Architect-Engineer  Contracting.  Topics 
covered  include  the  following: 

Subject  Area  1:  Mission  and  structure  of  the  USACE 

Project  management  business  process 
Civil  and  military  construction 
Funding 

Subject  Area  2:  Acquisition  and  source  selection  planning 

Differences  in  construction  contracting 
Solicitations 
Funding  and  awarding 
Procurement 

Subject  Area  3:  Briefings 

Contract  administration 
Contract  changes 
Claims  and  audits 
Closeout  and  terminations 

Subject  Area  4:  Authority  and  selection 

Solicitation,  negotiation,  and  award 
Task  orders 
A-E  post  award 

A  total  of  45  tasks  distributed  across  the  four  subject  areas  comprise  the  51C/1102 
Proficiency  Guide  (USACE,  2011a).  Included  at  the  end  of  each  task  are  performance 
measures  enabling  the  mentor/supervisor  to  confirm  understanding  of  details  discussed 
inside  each  task.  Utilizing  the  51C/1102  Proficiency  Guide's  (USACE,  2011a)  structure 
of  tasks  and  incorporating  significant  performance  measures  became  the  basis  for  the 
research  team’s  PAT  content  and  order.  Mirroring  the  51C/1102  Proficiency  Guide 
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simplifies  the  mentor/leader’s  supervision  of  a  51C/1102s  training  progress  and  ensures  a 
comprehensive  exposure  to  subject  matter  in  a  more  structured  or  formulated  manner. 

In  order  to  develop  a  PAT,  the  research  team  needed  considerable  understanding 
of  the  principles  of  test  construction.  The  research  team  conducted  a  literature  review  of 
academic  and  theoretical  principles,  as  well  as  military  doctrine  applicable  to  developing 
the  PAT.  This  knowledge  provided  the  research  team  with  the  tools  to  create  a  sound  and 
credible  PAT  for  assisting  in  the  development  of  contracting  officers  inside  the  USACE. 

E.  SUMMARY 

In  the  literature  review,  the  research  team  conducted  a  comprehensive  fact-finding 
mission  to  discover  the  most  effective  means  of  measuring  a  test  taker’s  proficiency  in  a 
testable  subject.  The  research  and/or  theory  by  experts  in  the  fields  of  education  and  test 
construction,  as  well  as  the  professional  report  by  Arzu  et  al.  (2010),  established  the 
structure,  format,  and  means  of  proofing  the  reliability  of  the  proposed  PAT.  This 
enabled  the  research  team  to  begin  gathering  subject  matter  materials  for  test-item 
construction. 

The  complex  and  challenging  subject  of  construction  contracting  is  an  important 
facet  of  the  USACE  mission.  The  unique  unit  structure  of  the  USACE  required  the 
research  team  to  explore  the  NCO  and  its  context  within  the  USACE.  In  Chapter  III,  the 
research  team  describes  in  detail  the  NCO  structure  and  strategy. 
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III.  USACE  NATIONAL  CONTRACTING  ORGANIZATION  AND 
THE  51C/1102  PROFICIENCY  GUIDE 

A.  INTRODUCTION 

This  chapter’s  purpose  is  to  provide  an  overview  of  the  USACE  NCO  and 
implementation  of  the  51C/1102  Proficiency  Guide  for  Construction,  Architect-Engineer 
and  Contingency  Contracting  (USACE,  2011a).  The  NCO  overview  will  include  the 
organization’s  structure,  command  relationship  to  USACE,  and  training  methodology. 
The  training  methodology  includes  implementation  of  the  51C/1102  Proficiency  Guide 
(USACE,  2011a),  which  is  discussed  in  detail,  because  this  guide  is  the  most  significant 
resource  for  this  project. 

B.  ESTABLISHMENT  OF  THE  U.S.  ARMY  CORPS  OF  ENGINEERS 

NATIONAL  CONTRACTING  ORGANIZATION 

The  NCO  was  established  as  a  result  of  a  USACE  organizational  structure  review 
conducted  in  2007  to  improve  the  effectiveness  of  the  contracting  mission  within  USACE 
(Van  Antwerp,  2010).  The  vision  of  the  NCO  is  “a  GREAT  engineering  force  of  highly 
disciplined  people  working  with  our  partners  through  disciplined  thought  and  action  to 
deliver  innovative  and  sustainable  solutions  to  the  Nation’s  engineering  challenges” 
(USACE,  2011a).  The  stated  mission  of  the  NCO  is  to  “provide  vital  public  engineering 
services  in  peace  and  war  to  strengthen  our  Nation’s  security,  energize  the  economy,  and 
reduce  risks  from  disasters”  (USACE,  2011a).  The  NCO  contracting  workforce  is 
composed  of  over  1,500  personnel,  including  1,300  contracting  officers  and  contract 
specialists  (Van  Antwerp,  2010).  The  NCO  was  established  in  the  same  year  that  the 
Gansler  Commission  report  of  2007  required  urgent  reform  on  the  part  of  the  Army.  As 
discussed  in  Chapter  II,  the  Gansler  report  detailed  systemic  failures  within  the  Army, 
particularly  for  expeditionary  contracting  operations.  The  establishment  of  the  NCO  was 
consistent  with  recommendations  from  the  Gansler  report  to  provide  for  a  single 
command  structure  for  the  contracting  workforce  within  the  USACE  (Van  Antwerp, 
2010). 
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C.  NCO  ORGANIZATIONAL  STRUCTURE 

Figure  12  reflects  the  established  organizational  structure  for  the  NCO. 


Figure  12.  NCO  Organizational  Structure 
(From  US  ACE,  201  lb,  slide  6) 


As  shown  in  Figure  12,  the  director  of  the  NCO  is  a  senior  executive  service 
(SES)  position  that  has  an  equivalency  of  a  two-star  general  position.  As  a  result  of  the 
establishment  of  the  NCO,  the  number  of  principle  assistants  responsible  for  contracting 
(PARCs)  increased  from  one  to  three  to  allow  greater  oversight  of  regional  areas  (Van 
Antwerp,  2010).  Further  leadership  was  implemented  by  creating  regional  and  center 
contracting  chiefs  for  all  of  the  USACE  Districts  and  Centers  (Van  Antwerp,  2010). 

The  command  and  support  relationships  between  the  NCO  and  USACE  are 
illustrated  in  Figure  13. 
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Functionally  Independent  but  Operationally  Integrated! 
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______  Support  Relationship 

Figure  13.  NCO  Alignment  Within  the  USACE 
(From  USACE,  2011b,  slide  5) 

Figure  13  illustrates  the  leadership  command  and  support  levels  within  the 
USACE.  Under  this  structure,  the  NCO  has  command  authority  over  the  regionally 
aligned  centers  and  districts  that  provide  support  to  their  respective  divisions  and  districts 
within  the  USACE.  The  funded  line  reflects  that  regional  organizations  and  above  are 
direct  funded,  while  districts  are  supported  through  funded  projects.  The  effect  is  that  the 
size  of  the  district  contracting  personnel  pool  varies  based  on  the  number  of  projects. 

Within  this  organizational  structure,  the  NCO  has  further  established  MCCTs  to 
provide  support  for  military  expeditionary  contracting  operations.  The  NCO  currently  has 
nine  MCCTs  that  are  aligned  with  CONUS  districts  with  a  few  more  identified  to  stand 
up  over  the  next  few  fiscal  years.  The  selected  districts  have  a  military  mission  and  are 
able  to  train  and  prepare  the  soldiers  assigned  to  the  teams  to  be  proficient  in  contingency 
construction  and  Architect-Engineer  contracting  prior  to  deployment.  The  structure  of  the 
MCCTs  typically  consists  of  four  MCCOs  designated  with  the  5 1C  MOS  code;  these 
MCCOs  include  one  lieutenant  colonel,  one  major,  and  two  senior  enlisted  non¬ 
commissioned  officers  at  the  master  sergeant  and  sergeant  first-class  ranks.  This  structure 
closely  resembles  the  ECC’s  CCT  model  with  the  exception  that  the  USACE  NCO  rank 
structure  is  typically  one  grade  higher  at  each  position  than  the  ECC  model.  To  assist 
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the  MCCT-aligned  districts,  supervisors,  and  trainers,  the  USACE  NCO  developed  the 
51  C/1 102  Proficiency  Guide  For  Construction,  Architect-Engineer  and  Contingency 
Contracting  (USACE,  2011a). 

D.  TRAINING  STRATEGY 

The  USACE  NCO  training  strategy  is  to  develop  MCCOs  and  newly  assigned 
1102  civilians  with  distinctive  USACE  contracting  skills  at  an  acceptable  level  within  the 
first  year  to  18  months.  This  allows  MCCOs  to  support  expeditionary  deployments  and 
other  broadening  opportunities  within  the  remaining  timeframe  of  a  two-  to  three-year 
assignment.  The  USACE  contracting- specific  training  supplements  training  required  for 
Defense  Acquisition  Workforce  Improvement  Act  (DAWIA)  certification  levels  that 
require  DAU  or  equivalent  classes,  as  well  as  civilian  business-related  college  credits.  It 
is  expected  that  MCCOs  that  are  not  Level  II  certified  upon  arrival  to  the  unit  achieve 
Level  II  certification  by  the  end  of  their  assignment  with  the  USACE.  In  addition  to 
contracting-related  training,  MCCOs  must  also  adhere  to  Army-specific  training.  For 
example,  training  will  include  Army  physical  fitness  standards,  marksmanship,  pre¬ 
deployment  training,  and  other  annual  training  requirements  identified  in  Army 
Regulations,  such  as  AR  350-1  Army  Training  and  Leader  Development  (DoA,  2009). 
For  the  purposes  of  this  report,  the  USACE  contracting- specific  training  is  discussed 
further. 

To  develop  the  unique  skill  sets  required  of  an  MCCO  and  1102,  the  USACE  has 
established  formal  training  courses  and  identified  key  and  supporting  individual 
contracting  tasks  to  train  through  the  51C/1102  Proficiency  Guide  (USACE,  2011a).  A 
5 1C  MOS  soldier  will  receive  basic  courses  in  contracting  through  the  DAU  or 
equivalent  institutional  training  to  meet  DAWIA  certification  standards.  In  addition  to 
these  courses,  the  USACE  has  established  its  own  educational  system  known  as  the  ULC 
that  offers  “virtual  engineering  and  mission  support  training  to  develop  and  sustain 
competencies  cultivating  a  competent,  disciplined,  resilient  workforce  that  provides 
quality  solutions”  (USACE,  2011a).  Additionally,  the  USACE  offers  PROSPECT  to 
develop  unique  skills  of  the  workforce  (USACE,  2011).  There  are  five  required  courses 
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identified  as  A-E  Contracting,  Construction  Contract  Administration,  Estimating  for 
Construction  Contract  Modifications,  Negotiating  Construction  Contract  Modifications, 
and  District  Officer  Introductory  Course,  as  well  as  highly  recommended  courses  to 
include  Design/Build  Construction  (USACE,  2011a).  There  are  a  multitude  of  common 
and  specialized  tasks  expected  of  a  USACE  contracting  officer  throughout  the 
contracting  process.  In  addition  to  formal  DAU  and  PROSPECT  training,  the  USACE 
(2011a)  has  developed  a  51C/1102  Proficiency  Guide  for  key  and  supporting  skill  tasks 
required  of  a  USACE  contracting  officer. 

Many  of  the  key  and  supporting  skill  tasks  identified  in  the  USACE  51C/1102 
Proficiency  Guide  (2011a)  supplement  the  tasks  that  can  be  found  in  the  Army 
Contracting  Command  51C  Level  One  Proficiency  Guide  (DoA,  2010).  The  ACC 
proficiency  guide  is  used  to  establish  an  acceptable  level  of  individual  competency  in  the 
focus  areas  of  commercial  items,  simplified  acquisition  procedures,  procurement  of 
supplies,  services  and  minor  construction,  contract  planning,  and  contract  closeout  (DoA, 
2010).  These  skills  are  contracting  basics  that  are  also  applicable  to  USACE  contracting 
officers.  Therefore,  it  is  desirable  that  MCCOs  assigned  to  the  USACE  have  at  least  one 
previous  assignment  with  the  ACC  prior  to  being  assigned  to  USACE,  because  the 
specialized  tasks  require  training  at  a  more  advanced  or  specialized  level  (USACE, 
2011a).  Due  to  Army  manning  constraints,  it  is  not  always  possible  to  assign  MCCOs  to 
the  USACE  that  have  had  a  previous  assignment  with  the  ACC.  Therefore,  the  training 
strategy  is  adapted  to  assign  personnel  within  the  USACE  to  have  a  mix  of  “seasoned,”  or 
experienced,  MCCOs  with  initial  entry  personnel. 

The  initial  entry  personnel  to  the  5 1C  field  will  focus  on  areas  commonly 
associated  with  ACC  training  for  their  first  six  months  and  transition  to  USACE 
construction  and  A-E  contracting  tasks  for  their  remaining  time.  Figure  14  illustrates  the 
typical  training  timeline  for  a  USACE  MCCO. 
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Initial  6  Months 


1st  Year 


1-1.5  Year 


1.5 -3rd  Year 


Figure  14.  USACE  5 1C  Training  and  Developmental  Timeline 
(From  USACE,  2011a,  ch.  1  p.  4) 


Construction  and  A-E  contracting  tasks  are  the  focus  of  the  first  year  of  an  MCCO 
assignment  with  a  slight  exception  if  this  is  the  initial  5 1C  contracting  assignment  for  the 
MCCO.  Newly  assigned  MCCOs/1102s  are  assigned  supervisors  and  mentors  to  assist 
them  in  their  development  of  and  training  in  contracting-related  tasks.  MCCOs  in  their 
first  year  are  typically  non-deployable  for  contracting  contingency  missions  due  to  their 
lack  of  experience,  which  is  supported  by  the  USACE  and  guidance  from  the  Director  of 
Acquisition  Career  Management  (DoA,  2008).  MCCOs  are  assessed  by  their 
mentors/supervisors  as  their  training  progresses  and  should  meet  acceptable  standards 
within  12  to  18  months.  After  the  initial  12-  to  18-month  training  period,  officers  become 
part  of  an  availability  pool  to  support  expeditionary  operations  that  are  typically  six- 
month  deployment  rotations.  Trained  MCCOs  may  also  have  the  opportunity  for  60-  to 
120-day  temporary  assignments  to  other  districts  or  missions  to  broaden  their  contracting 
experience.  For  MCCOs  and  1102s  that  have  completed  the  initial  level  of  training, 
continuous  learning  is  emphasized  through  individual  planning  with  supervisor  approval 
that  requires  80  continuous  learning  points  (CLPs)  every  two  years  that  is  consistent  with 
acquisition  workforce  policy. 

E.  PROFICIENCY  GUIDE 

The  USACE  51C/1102  Proficiency  Guide  (2011a)  is  intended  primarily  for 
MCCOs  and  the  supervisors  and  mentors  assigned  to  them,  but  the  guide  may  also  be 
used  for  civilians,  such  as  interns.  As  previously  discussed,  the  51C/1102  Proficiency 
Guide  (USACE,  2011a)  is  not  intended  to  replace  ACC-level  tasks,  but  it  is  designed  in 
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addition  to  these  tasks  to  focus  on  the  USACE’s  unique  skill  requirements  for  a 
contracting  professional.  The  51C/1102  Proficiency  Guide  (USACE,  2011a)  has  45 
specific  tasks  identified  overall  that  are  broken  down  into  four  subject  areas  as  follows: 

•  Subject  Area  1:  USACE 

•  Subject  Area  2:  Contracting  Pre-Award  Tasks  for  Construction  Contracting 

•  Subject  Area  3:  Contracting  Post-Award  Tasks  for  Construction  Contracts 

•  Subject  Area  4:  Architect-Engineer  Contracting 

Appendix  C  provides  a  breakdown  of  the  associated  specific  tasks  related  to  each  subject 
area. 

The  51  C/1 102  Proficiency  Guide  (USACE,  20 1 1  a)_is  structured  based  on  Army 
regulations  such  as  AR  350-1  (DoA,  2009)  that  dictate  education  and  training 
publications  “establish  tasks,  conditions,  and  standards  for  military  occupational 
specialties.”  Accordingly,  each  task,  the  51C/1102  Proficiency  Guide  (USACE,  2011a) 
provides  the  conditions  that  the  task  is  meant  to  be  performed  under  and  the  acceptable 
level  of  performance  indicated  through  the  standard  and  associated  performance  steps 
that  are  rated  on  a  subjective  “GO”  or  “NO  GO”  basis  by  the  evaluator.  The  task, 
conditions,  and  performance  measures  give  focus  for  the  PAT  and  scenario  development 
and  provide  the  research  team  with  key  areas  to  emphasize. 

F.  SUMMARY 

In  this  chapter,  the  research  team  provided  an  overview  of  the  USACE  NCO  and 
implementation  of  the  51  C/1 102  Proficiency  Guide  for  Construction ,  Architect-Engineer, 
and  Contingency  Contracting  (USACE,  2011a).  The  NCO  organizational  structure, 
command  relationship  to  the  USACE,  and  training  methodology  were  also  described 
down  to  the  MCCO  individual  level  that  the  PAT  is  being  developed  for.  The  51C/1102 
Proficiency  Guide  task,  conditions,  standards,  and  proficiency  measures  were  discussed 
that  provides  the  authors  a  focus  for  PAT  development. 

MCCTs  within  the  USACE  NCO  structure  are  a  relatively  new  concept  and 
have  been  expanding  based  on  additional  resources  and  funding  as  a  result  of 
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recommendations  by  the  Gansler  (2007)  Commission  under  the  Army’s  “Grow  the 
Acquisition  Workforce”  initiative  (Van  Antwerp,  2010).  To  train  the  MCCOs  assigned  to 
the  MCCTs,  the  USACE  NCO  has  adopted  a  similar  training  strategy  to  that  of  the  ACC 
by  developing  its  own  version  of  a  Soldier ’s  Manual  of  Common  Tasks  (SMCT)  that  is 
concentrated  on  USACE  construction  and  Architect-Engineer  contracting  functions.  As 
Arzu  et  al.  (2010)  determined,  the  ACC  SMCT  was  “incomplete  without  a  formalized 
proficiency  assessment  test”  (p.  59),  and  so  too  is  the  USACE  51  C/1 102  Proficiency 
Guide.  The  ECC  has  successfully  implemented  the  PAT  with  supporting  foundational 
work  by  Arzu  et  al.  (2010).  Due  to  the  success  within  the  ACC,  the  USACE  (2011a)  has 
requested  the  support  of  the  authors  to  develop  a  PAT  based  on  the  tasks  identified  in 
their  51C/1102  Proficiency  Guide ,  which  is  the  primary  goal  of  this  research.  The 
development  of  the  USACE  PAT  is  discussed  in  the  following  chapter. 
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IV.  PROFICIENCY  ASSESSMENT  TEST  DEVELOPMENT 


A.  INTRODUCTION 

In  this  chapter,  the  researchers  discuss  the  development  of  the  PAT.  The  PAT  was 
developed  through  a  combination  of  customer  input,  as  well  as  techniques  and  practices 
learned  through  the  literature  review.  The  test  construction  plan,  test  blueprint,  and  item 
writing  are  discussed  in  further  detail.  To  determine  validity  of  the  research  approach,  the 
team  developed  a  pilot  PAT  that  was  administered  to  experienced  and  knowledgeable 
MCCOs  within  USACE  that  provided  anonymous  feedback. 

B.  TEST  CONSTRUCTION  PLAN 

In  constructing  the  PAT,  the  research  team  applied  Downing  and  Haladyna’s 
(2006)  12  steps  for  effective  test  development  identified  in  the  literature  review.  The  12 
steps  include  overall  plan,  content  definition,  test  specifications,  item  development,  test 
design  and  assembly,  test  production,  test  administration,  scoring  test  responses,  passing 
scores,  reporting  test  results,  item  banking,  and  test  technical  report  (Downing  & 
Haladyna,  2006). 

Step  1  is  the  overall  plan,  which  encompasses  “systematic  guidance  for  all  test 
development  activities”  (Downing  &  Haladyna,  2006).  This  step  is  the  basis  for  most  of 
the  research  scope  and  deliverables  described  in  Chapter  I.  The  research  team  developed 
a  PAT  for  measuring  51C/1102  proficiency  tasks  in  a  format  that  enables  ease  of  use  in 
either  a  written  or  computerized  format.  Developed  scenarios  and  items  were  vetted 
through  feedback  from  USACE  subject  matter  experts.  The  research  team  also  conducted 
a  pilot  PAT  for  further  assurance  that  the  approach  met  the  requested  time  and  difficulty 
expectations  expressed  by  the  USACE.  The  USACE  is  the  decision-maker  and  will  best 
determine  how  to  implement,  administer,  safeguard,  and  update  the  delivered  test  upon 
completion  of  our  PAT  deliverable. 

Content  definition  (Step  2)  asks  the  question,  “What  content  is  to  be  tested?” 

(Downing  &  Haladyna,  2006).  The  research  team,  with  guidance  from  the  USACE, 

treated  everything  in  the  USACE  51C/1102  Proficiency  Guide  (2011a)  as  acceptable  for 
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testing.  Additionally,  any  references  listed  for  proficiency  tasks  were  also  determined 
acceptable  for  item  writing.  References  not  listed  by  the  51C/1102  Proficiency  Guide 
were  for  the  most  part  restricted  from  being  utilized  for  item  writing  as  the  research  team 
did  not  have  adequate  experience  or  domain  knowledge  to  ascertain  how  the  reference 
impacted  the  organization. 

Test  specifications  and  item  development,  Steps  3  and  4,  respectively,  are 
discussed  in  detail  in  Sections  C  and  D  of  this  chapter  (Downing  &  Haladyna,  2006). 

For  test  design  and  assembly  (Step  5),  the  research  team  designed  the  test  to 
easily  facilitate  incorporation  into  a  computer-based  delivery  method.  The  items  written 
for  the  PAT  are  in  the  form  of  single -best-answer  multiple-choice  questions,  which 
facilitates  a  simplified  answer  key  (Downing  &  Haladyna,  2006).  The  simple  answer  key 
allows  for  easier  test  administration,  as  well  as  instant  grading  and  feedback  to  the  test¬ 
taking  individual.  The  research  team  assembled  and  grouped  items  based  on  the  45 
identified  tasks  in  the  51C/1102  Proficiency  Guide.  The  research  team  checked  to  ensure 
that  the  answers  on  the  final  PAT  deliverable  were  balanced  as  far  as  where  the  correct 
response  was  indicated. 

Test  production  concerns  in  Step  6  are  the  security  and  safeguarding  of  the  PAT 
deliverable  (Downing  &  Haladyna,  2006).  To  address  this  concern,  the  research  team 
worked  exclusively  through  the  designated  USACE  representatives.  The  USACE  is 
aware  of  the  need  to  safeguard  the  PAT  and  determine  the  extent  that  items  are  reviewed 
for  validity. 

The  concerns  for  Step  7,  test  administration,  are  the  security  issues  associated 
with  a  decentralized  computer  delivery  method  (Downing  &  Haladyna,  2006).  The 
research  team  has  taken  action  to  allow  flexibility  in  test  administration  by  providing  an 
overage  of  items  to  allow  for  an  item  bank  grouped  by  task.  The  item  bank  will  make 
administering  different  versions  of  the  PAT  possible  and  allow  that  some  test  items  that 
are  determined  to  be  highly  unreliable  or  misunderstood  be  thrown  out. 
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Scoring  examination  responses  is  Step  8  (Downing  &  Haladyna,  2006).  As 
discussed,  the  research  team  designed  the  test  for  extremely  simple  scoring  that  can  be 
easily  accomplished  through  computer  software  that  provides  immediate  feedback  to  the 
test  administrator  and  the  examinee. 

The  passing  score  (Step  9)  discussed  between  the  USACE  and  the  research  team 
was  determined  to  be  answering  80%  or  more  of  the  items  correctly  (Downing  & 
Haladyna,  2006).  This  is  a  traditional  standard  and  allows  the  examinee  some  room  for 
error.  The  USACE  always  has  the  ability  to  raise  or  lower  the  standard  for  passing  as 
PAT  reliability  becomes  more  greatly  defined.  A  passing  score  represents  a  “GO”  or  “NO 
GO”  for  the  given  PAT. 

Reporting  examination  results  (Step  10),  item  banking  (Step  11),  and  test 
technical  report  (Step  12)  are  areas  that  are  beyond  the  scope  of  this  research.  The 
research  team  recommends  that  the  USACE  provide  accurate  and  timely  feedback  to  the 
examinee  upon  PAT  submission.  Given  the  test  design  and  simplified  administration, 
prompt  feedback  to  examinees  should  not  be  a  problem  to  implement.  As  discussed,  the 
USACE  should  consider  item-banking  approaches  to  protect  the  security  of  the  PAT. 
Whether  USACE  determines  to  outsource  test  administration  through  a  contract  or 
conduct  administration  internally,  a  test  technical  report  will  be  critical  to  indicate  PAT 
validity  and  recommendations  to  improve  future  versions. 

C.  TEST  BLUEPRINT 

The  research  team  developed  a  test  blueprint  for  the  PAT.  The  general  process  of 
creating  a  test  blueprint  is  described  in  the  literature  review.  Test  blueprint  is  Step  3,  test 
specifications,  of  Downing  and  Haladyna’s  (2006)  12  steps  for  effective  test 
development.  Test  specifications  or  blueprint  development  by  the  research  team  was  the 
final  part  of  operational  planning  for  the  PAT  (Downing  &  Haladyna,  2006). 

The  research  team  chose  the  selected  response  approach  as  a  testing  format.  The 
rationale  behind  this  decision  was  to  meet  the  customer’s  expectation  for  test 
administration  that  allows  flexibility  and  to  follow  the  precedent  set  by  the  ACC’s  PATs 
for  contingency  contracting.  The  selected  response  approach  simplifies  the  administration 
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requirements  for  the  PAT  that  is  desired  given  the  geographic  separation  within  the 
US  ACE  command.  The  PAT  is  to  be  used  as  a  supplement  to  the  51  C/1 102  Proficiency 
Guide  (US ACE,  2011a)  that  allows  trainers  and  supervisors  to  add  performance  and 
constructed  response  approaches  for  the  center/district  requirements  to  further 
demonstrate  proficiency  for  a  given  task  area. 

The  test  blueprint  the  research  team  developed  in  collaboration  with  USACE 
determined  that  each  PAT  administered  would  consist  of  20  items.  This  number  was  to 
account  for  reliability  as  well  as  time  constraints.  Twenty  questions  are  also  consistent 
with  the  precedent  set  by  the  testing  administration  conducted  by  the  ECC  contingency 
contracting  PATs.  Each  test-item  format  is  in  the  form  of  single -best  answer  from 
multiple-choice  selections.  The  research  team  classified  questions  into  three  categories 
modified  from  Bloom’s  Taxonomy  of  Educational  Objectives:  Cognitive  Domain  (1956). 
The  categories  include  recall,  comprehension,  and  application.  Based  on  the  research 
team’s  limited  knowledge  and  experience  in  construction  and  architect-engineer 
contracting,  the  overwhelming  majority  of  questions  the  research  team  produced  fell  into 
the  recall  category.  The  research  team  relied  on  assistance  from  the  USACE  for 
clarification  of  the  reference  material,  as  well  as  novel  scenarios  and  input  for 
comprehension  and  application  questions. 

The  blueprint  the  research  team  determined  was  to  concentrate  questions  covering 
the  major  performance  measures  of  each  task  with  a  limited  number  of  questions  based 
on  minor  areas  taken  from  references  associated  with  the  task.  The  research  team 
developed  questions  with  the  intent  of  allowing  an  examinee  approximately  two  minutes 
per  question,  regardless  of  the  category  type.  All  questions  developed  would  be  given 
equal  scoring  and  examinees  would  be  allowed  to  utilize  any  reference  material  identified 
in  the  51C/1102  Proficiency  Guide  (USACE,  2011a)  while  testing.  The  consequences  of 
an  examinee  failing  the  PAT  were  determined  to  be  low  as  the  examinee  could  retake  the 
PAT  with  minor  impact  on  recurring  test  administration,  time  and  costs. 
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D.  ITEM  WRITING 


The  test  blueprint  that  the  research  team  developed  determined  the  PAT  to  be 
constructed  as  a  selected  response  approach  consisting  of  single-best  answer  from  a 
multiple-choice  answer  selection.  The  research  team  constructed  items  that  utilized  the 
principles  and  guidelines  discussed  in  Chapter  II  of  this  report  that  pertained  to 
developing  multiple-choice  items. 

A  sample  of  items  that  the  research  team  developed  can  be  found  in  Appendix  D. 
Figure  15  is  an  example  that  demonstrates  the  strategies  learned  for  developing  multiple- 
choice  items. 


Which  of  the  following  BEST  illustrates  military  construction? 
r* 

A.  Highways  and  bridges 

p 

B.  Hydroelectric  power  facilities  and  water  supply 

C.  Housing  and  waste  management  facilities 
r 

D.  Wastewater  treatments  and  dredging 
Answer  Key:  C 

Figure  15.  Item  Constructed  for  Task  1-4  of  the  USACE  51C/1102  Proficiency  Guide 

(From  US  ACE,  2011a) 

As  shown  in  Figure  15,  the  research  team  applied  the  Downing  and  Haladyna 
(2006)  strategies  and  rules.  The  strategies  the  research  team  incorporated  for  constructing 
this  particular  item  included  positively  wording  the  item  stem,  reducing  the  reading 
involved  by  the  examinee,  ensuring  only  one  correct  answer,  placing  the  answer 
selections  in  a  reasonable  order,  removing  superfluous  language,  and  staying  away  from 
content  that  is  too  specific  (Downing  &  Haladyna,  2006).  The  item  developed  is  also 
consistent  with  rules  for  developing  effective  multiple-choice  items:  the  composition  of 
the  item  is  straightforward,  the  distractors  are  effective,  only  one  answer  is  correct,  the 
item  does  not  give  away  hints  for  other  items  and  vice  versa,  and  the  content  is  not  trivial 
(Downing  &  Haladyna,  2006). 
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The  example  in  Figure  15  and  the  other  examples  provided  in  Appendix  D  are 
representative  of  typical  multiple -choice  items  the  research  team  developed.  The  research 
team  utilized  item-writing  references  to  gain  insight  for  developing  effective  multiple- 
choice  items. 

E.  RELIABILITY  AND  VALIDITY 

The  research  team  identified  reliability  and  validity  as  core  test  principles  during 
the  literature  review.  Reliability  involves  the  degree  that  examinees  achieve  consistent 
results  on  the  PAT  (Marshall  &  Hales,  1972).  Content  validity  is  a  measure  of 
acceptability  with  which  a  body  of  intellectual  content  is  covered  (Nunnally  &  Ator, 
1972).  To  ensure  that  items  relate  to  important  task  content,  the  research  team 
constructed  items  to  reinforce  the  standards  and  performance  tasks  associated  with  a 
given  task.  The  standards  and  performance  tasks  have  already  been  pre-determined  as 
areas  to  show  proficiency  in  for  a  given  task  as  publicized  in  the  51  C/1 102  Proficiency 
Guide  (USACE,  2011a).  By  covering  these  areas,  the  research  team  is  reasonably 
confident  that  as  long  as  the  items  have  a  high  degree  of  reliability,  they  will  also  have 
validity. 

Ascertaining  the  reliability  for  the  PAT  as  described  in  the  literature  review  is 
beyond  the  scope  of  the  research.  Determining  reliability  through  methods  such  as 
test/retest,  parallel-forms,  alternate  forms,  and  internal-consistency  estimates  are  areas  for 
further  research.  Alternatively,  the  research  team  implemented  a  pilot  PAT  for 
experienced  MCCO’s  within  the  USACE.  The  PAT  results  did  not  show  statistically 
significant  data  to  draw  any  conclusion  of  reliability.  The  research  team  did  receive 
valuable  feedback  from  examinees  through  an  anonymous  survey  following  the  pilot 
PAT,  which  provided  the  research  team  insight  for  blueprinting  as  well  as  some 
assurance  that  the  items  were  challenging  with  a  sound  methodology.  Although  the 
USACE  has  vetted  the  items  the  PAT  incorporates,  the  research  team  recommends  that 
the  USACE  take  future  action  to  substantiate  the  PAT  as  reliable  and  valid  through 
analysis  of  PAT  responses  during  implementation. 
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F. 


PILOT  PAT  FEEDBACK 


The  pilot  PAT  the  research  team  developed  and  deployed  provided  an  anonymous 
link  consisting  of  25  multiple-choice  items  with  a  time  limit  of  60  minutes  offered  to  all 
MCCOs  within  the  USACE  who  had  sufficient  experience.  As  indicated  in  the  preceding 
section,  the  research  team  did  not  have  statistically  significant  data  to  draw  a  conclusion 
on  reliability.  The  research  team  further  developed  an  anonymous  survey  link  for 
examinees  to  provide  feedback  on  the  PAT. 

The  feedback  provided  by  the  examinees  was  limited  but  useful  to  the  research 
team.  The  consensus  among  those  examinees  that  provided  feedback  indicated  that 
approximately  2.5  minutes  per  question  was  adequate;  however,  all  respondents  had  a 
preference  for  the  exam  to  be  untimed.  All  respondents  confirmed  that  the  pilot  PAT 
properly  captured  the  subject  matter  of  the  51C/1102  Proficiency  Guide  (USACE,  2011a) 
and  was  challenging  to  them.  Items  for  the  pilot  PAT  were  developed  using  both  the 
51C/1102  Proficiency  Guide  and  the  associated  references  linked  from  the  51C/1102 
Proficiency  Guide  for  the  specified  tasks.  All  respondents  felt  the  subject  matter  for  the 
PAT  should  not  be  restricted  solely  to  the  51C/1102  Proficiency  Guide  but  should  also 
include  the  reference  material  to  promote  learning  and  research. 

Responses  to  a  question  asking  respondents  to  identify  a  negative  aspect  of  the 
pilot  test  produced  limited  responses  with  the  take-away  for  the  research  team  that  not  all 
examinees  completed  the  exam  within  the  allotted  time.  For  time  purposes,  the  research 
team  needs  to  ensure  that  reading  is  reduced  for  items  and  that  superfluous  language  is 
removed.  Providing  more  or  less  time  for  the  PAT  should  be  relatively  easy  to  implement 
as  needed  through  analysis  of  future  PAT  responses. 

G.  SUMMARY 

The  development  of  the  PAT  consisted  of  a  combination  of  customer  input,  as 
well  as  techniques  and  practices  learned  through  the  literature  review.  The  research  team 
applied  Downing  and  Haladyna’s  (2006)  12  steps  for  effective  test  development  in 
determining  the  PAT  construction  plan.  The  test  blueprint  is  a  selected  response  approach 
of  a  single -best  answer  from  a  multiple-choice  selection.  The  blueprint  selected  allows 
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flexibility  and  follows  the  precedent  set  by  the  ACC’s  PATs  for  contingency  contracting. 
The  research  team  gained  insight  through  implementing  a  pilot  PAT.  The  examinees  for 
the  pilot  PAT  were  anonymous  MCCOs  within  USACE  who  had  sufficient  experience 
and  knowledge.  The  MCCOs  provided  feedback  that  further  assured  the  research  team 
that  the  PAT  methodology  is  sound.  Chapter  V  summarizes  the  research,  provides  the 
conclusion,  and  recommends  areas  for  further  research. 
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V.  SUMMARY,  CONCLUSION,  AND  AREAS  FOR 
FURTHER  RESEARCH 

A.  SUMMARY 

The  US  ACE  is  a  vital  component  of  U.S.  national  defense  that  executes  necessary 
construction  contracts  in  civil,  military,  and  contingency  scenarios.  Near  the  epicenter  of 
responsibility  are  contracting  officers  who  ensure  the  USACE  mission  happens  with  the 
greatest  of  efficiency.  Minimal  knowledge  of  construction  contracting  is  imparted  to  an 
Army  5 1C  contracting  officer  before  being  assigned  to  the  USACE,  where  it  becomes  a 
primary  function  of  the  contracting  officer.  In  addition,  the  DoDIG  has  identified 
management  oversight,  control,  and  policy  enforcement  as  contracting  deficiencies,  and 
thus  require  ongoing  attention  in  an  effort  to  protect  valuable  taxpayer  funds  (DoDIG, 
2009).  In  2011,  the  USACE  produced  the  51  C/1 102  Proficiency  Guide  (USACE,  2011a) 
in  an  effort  to  define  training  tasks  for  leaders  and  mentors  in  developing  neophyte 
contracting  officers  unfamiliar  with  construction  contracting. 

Following  the  example  of  the  Arzu  et  al.  (2010)  research  team  developing  a  PAT 
for  the  ACC,  the  research  team  of  this  report  researched  and  created  a  similar  PAT  to 
assist  the  USACE  in  assessing  and  developing  the  proficiency  of  its  contracting  officers. 
The  theories  and  principles  reviewed  in  the  research  justified  the  composition  of  PAT 
items  and  provided  guidelines  for  valid  test-item  writing.  The  51C/1102  Proficiency 
Guide  (USACE,  2011a),  along  with  feedback  from  the  USACE  leadership,  organized  the 
test  items  into  45  tasks,  grouped  within  four  subject  areas,  which  provide  a  definitive  path 
for  contracting  officers  to  follow  for  professional  development  in  construction 
contracting. 

The  research  team’s  test  items  were  vetted  by  USACE  subject  matter  experts  for 
realism,  difficulty,  and  time  required  for  completing  the  test.  Averaging  20  questions  per 
task,  the  research  team  provided  a  written  PAT  containing  approximately  800  test  items 
covering  the  45  tasks  identified  in  the  51C/1102  Proficiency  Guide  (USACE,  2011a)  to 
the  USACE.  In  accordance  with  feedback  from  the  USACE,  test  items  were  arranged  into 
topical  groups,  as  shown  in  Figure  16. 
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SUBJECT  AREA  1:  US  ACE 

Tasks  1-1  to  1-3 
Tasks  1-4  to  1-6 

SUBJECT  AREA  2:  Pre-Award  Tasks 

Tasks  2-1  to  2-3 
Tasks  2-4  to  2-8 
Tasks  2-9  to  2-11 
Task  2-12 
Tasks  2-13  to  2-17 

SUBJECT  AREA  3:  Post-Award  Tasks 

Tasks  3-1  to  3-4 
Tasks  3-5  to  3-9 
Tasks  3-10  to  3-12 
Tasks  3-13  to  3-16 

SUBJECT  AREA  4:  A-E  Contracting 

Tasks  4-1  to  4-2 
Tasks  4-3  to  4-4 
Tasks  4-5  to  4-6 

Figure  16.  List  of  PAT  Test  Topics 


Arranging  test  items  into  14  topics  does  not  change  the  quantity  of  test  items  but 
provides  a  test  format  that  groups  tasks  into  14  tests  instead  of  45.  Utilizing  the  same 
battery  of  test  items,  this  arrangement  enlarges  the  pool  of  test  items  available  per  test. 
More  importantly,  this  arrangement  reduces  the  start  and  stop  time  needed  to  transition 
between  test-taking  sessions  while  maintaining  the  test’s  subject  assessment  of 
construction  contracting  tasks. 

B.  CONCLUSION 

The  researchers  of  this  project  have  explored  the  principles  of  test  development, 
theories  of  education  and  understanding,  and  strategic  business  models  on  the  importance 
of  resources.  The  intent  was  to  formulate  a  tool  for  the  USACE  to  improve  the  efficiency 
of  a  prized  resource:  its  contracting  officers.  The  research  team  produced  approximately 
800  questions  in  an  effort  to,  at  the  very  least,  establish  a  skill-testing  starting  point  for 
the  USACE  efforts  in  mentoring/developing  its  contracting  officers  new  to  construction 
contracting. 
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Although  the  research  team  created  and  utilized  a  pilot  PAT,  its  purpose  was  to 
quickly  and  easily  acquire  test  and  survey  data  for  research,  not  to  serve  as  an  official 
portion  of  the  PAT.  A  benefit  to  the  online  test  was  to  demonstrate  the  ease  of  creating 
and  introducing  test-taking  via  online  media  for  the  workforce.  Although  a  full-scale, 
online  PAT  was  beyond  the  scope  of  this  research,  the  online  format  provided  numerous 
benefits,  which  this  research  team  will  recommend  at  the  time  of  presentation  of  the 
written  PAT  to  the  USACE.  The  format  of  the  test  items  and  answers  in  the  PAT  can  be 
easily  converted  to  a  digital,  online  test  format. 

At  the  completion  of  this  research  project,  all  test  items  and  answers  were 
submitted  to  the  USACE.  The  USACE  will  have  ownership  of  the  written  PAT  test  items 
and  answers  that  the  research  team  created.  Afterwards,  the  USACE  can  convert  the  PAT 
from  its  written  format  into  a  usable  and  implementable  test.  The  PAT  will  allow  leaders 
in  the  USACE  to  more  accurately  measure  a  contracting  officer’s  knowledge  and 
capabilities  against  the  established  standards  contained  in  the  51C/1102  Proficiency 
Guide  (USACE,  2011a).  The  PAT  will  assess  a  contracting  officer’s  readiness  for 
positions  of  increased  responsibility  or  diversity  in  assignments. 

C.  AREAS  FOR  FURTHER  RESEARCH 

Several  topics  of  interest  were  discovered  during  the  past  year  that  the  research 
team  recommends  for  follow-on  research.  As  mentioned  in  Chapter  V,  Section  B, 
digitizing  the  PAT  and  formatting  it  for  online  testing  would  greatly  assist  the  USACE  in 
testing  its  workforce.  The  research  team  created  a  pilot  PAT  utilizing  the  Naval 
Postgraduate  School’s  SAKAI  web  server  to  host  the  test  in  a  format  utilized  for 
classroom  activities  as  well  as  course  exams.  Further  research  could  evaluate  formats  for 
online  tests  and/or  conduct  a  cost  analysis  of  possible  web  hosting  options  capable  of 
meeting  the  USACE  specifications/criteria. 

Limiting  the  written  PAT  to  20  questions  per  task  was  meant  to  enable  PAT 
creation  while  keeping  the  workload  reasonable  for  the  time  period  provided.  Creating 
additional  questions  to  increase  the  PAT  item  database  would  be  a  valuable  research 
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effort.  Also,  developing  additional  scenario-based  test  items  that  challenge  multiple  tasks 
and  skills  in  a  culminating  manner  would  greatly  increase  the  depth  of  proficiency 
challenged  by  the  PAT. 

Another  research  topic  could  be  to  expand  the  scope  of  the  PAT  to  other 
competencies  within  the  USACE  or  go  beyond  the  USACE  to  contracting  within  the 
Army.  A  research  team  could  construct  a  PAT  with  a  focus  on  another  contracting  or 
procurement  topic.  The  PAT  can  also  be  expanded  to  other  military  Agencies  or  the 
entire  DoD. 

A  vital  aspect  of  the  PAT  will  be  the  results.  A  research  team  could  assess  the 
results  from  the  USACE  PAT  to  inform  the  USACE  of  additional  training  needs.  This 
assessment  would  provide  essential  feedback  that  could  greatly  improve  training 
efficiency  within  the  USACE.  This  assessment  could  also  provide  feedback  to  DAU  for 
developing  DAU  courses. 

Finally,  in  Chapter  II,  Section  B,  the  research  team  reviewed  business  strategy  for 
competitive  advantage  and,  at  a  micro-level,  how  the  knowledge  and  experience  of 
contracting  officers  is  a  resource  to  achieve  competitive  advantage.  Another  topic  of 
research  could  be  the  analysis  at  a  macro-level  of  the  USACE  for  how  effectively  it  is 
utilizing  its  resources  (personnel  and  systems).  The  topic  has  tremendous  potential  to 
improve  the  strategic  and  competitive  advantage  of  the  USACE.  The  research  could 
develop  an  effective  means  of  measuring  the  performance  of  the  USACE  contracting  by 
utilizing  business  models  addressing  facets  of  efficiency,  structure,  and  value.  An 
example  could  be  evaluating  the  USACE’ s  use  of  resources  and  providing  feedback  that 
could  result  in  reforms  to  command  structure,  training  practices,  and/or  manpower  just  to 
mention  a  few  possibilities. 
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APPENDIX 


ESSENTIAL  PRINCIPLES  FOR  TEST  USERS 

1.  [A]  test  user  should  have  a  general  knowledge  of  measurement  principles 
and  of  the  limitations  of  test  interpretations,  and  how  well  they  match  the 
qualifications  required  to  use  a  specific  test. 

2.  A  test  user  should  know  his  own  qualifications  and  how  well  they  match 
the  qualifications  required  for  the  uses  of  specific  tests. 

3.  One  who  has  the  responsibility  for  decisions  about  individuals  or  policies 
that  are  based  on  test  results  should  have  an  understanding  of 
psychological  or  educational  measurement. 

4.  The  principal  test  users  within  an  organization  should  make  every  effort  to 
be  sure  that  all  those  in  the  organization  who  are  charged  with 
responsibilities  related  to  test  use  and  interpretation  have  received  training 
appropriate  to  those  responsibilities. 

5.  Anyone  administering  a  test  for  decision-making  purposes  should  be 
competent  to  administer  that  test  or  class  of  tests.  If  not  qualified,  he 
should  seek  the  necessary  training  regardless  of  his  educational 
attainments. 

6.  Tests  users  should  seek  to  avoid  bias  in  test  selection  and  interpretation  or 
even  the  appearance  of  discriminatory  practice. 

7.  Institutional  test  users  should  establish  procedures  for  periodic  internal 
review  of  test  use. 

8.  The  choice  or  development  of  tests,  test  batteries,  or  other  assessment 
procedures  should  be  based  on  clearly  formulated  goals  and  hypotheses. 

9.  The  test  user  should  consider  the  possibility  that  different  hypotheses  may 
be  appropriate  for  people  from  different  populations. 

10.  A  test  user  should  consider  more  than  one  variable  for  assessment  and  the 
assessment  of  any  given  variable  by  more  than  one  method. 

11.  In  choosing  a  method  of  assessment,  a  test  user  should  consider  his  own 
degree  of  experience  with  it  and  also  the  prior  experience  of  the  test  taker. 

12.  In  choosing  an  existing  test,  a  test  user  should  relate  its  history  of  research 
and  development  to  his  intended  use  of  the  instrument. 

13.  In  general,  a  test  user  should  try  to  choose  or  to  develop  an  assessment 
technique  in  which  “tester-effect”  is  minimized,  or  in  which  reliability  of 
assessment  across  testers  can  be  assured. 

14.  A  test  user  is  expected  to  follow  carefully  the  standardized  procedures 
described  in  the  test  manual  for  administering  a  test. 
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15.  A  test  user  must  fully  understand  the  administrative  procedures  to  be 
followed. 

16.  A  test  user  should  make  periodic  checks  on  material,  equipment  and 
procedures  to  maintain  standardization. 

17.  The  test  user  is  responsible  for  accuracy  in  scoring,  checking,  coding,  or 
record  test  results. 

18.  When  test  scoring  equipment  is  used  the  test  user  should  insist  on 
evidence  of  its  accuracy;  when  feasible,  he  should  make  spot  checks 
against  hand  scoring  or  develop  some  other  system  of  quality  control. 

19.  The  test  user  shares  with  the  test  developer  or  distributor  a  responsibility 
for  maintaining  test  security. 

20.  All  reasonable  precautions  should  be  taken  to  safeguard  test  material. 

21.  A  test  user  should  consider  the  total  context  of  testing  in  interpreting  an 
obtained  score  before  making  any  decisions. 

22.  A  test  user  should  recognize  that  estimates  of  reliability  do  not  indicate 
criterion-related  validity. 

23.  A  test  user  should  examine  carefully  the  rationale  and  validity  of 
computer-based  interpretations  of  test  scores. 

24.  In  norm-referenced  interpretations,  a  test  user  should  interpret  an  obtained 
score  with  reference  to  sets  of  norms  appropriate  for  the  individual  tested 
and  for  the  intended  use. 

25.  Test  users  should  avoid  the  use  of  terms  such  as  IQ,  IQ  equivalent,  or 
grade  equivalent  where  other  terms  provide  more  meaningful 
interpretations  of  a  score. 

26.  A  test  user  should  examine  differences  between  characteristics  of  a  person 
tested  and  of  those  of  the  population  on  whom  the  test  was  developed  or 
norms  developed.  His  responsibility  includes  deciding  whether  the 
differences  are  so  great  that  the  test  should  not  be  used  for  that  person. 

27.  The  test  user  should  consider  alternative  interpretations  of  a  given  score. 

28.  A  test  user  should  develop  procedures  for  systematically  eliminating  from 
data  filers  test-score  information  that  has,  because  of  the  lapse  of  time, 
become  obsolete. 
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B.  TWELVE  STEPS  FOR  EFFECTIVE  TEST  DEVELOPMENT 


Step  1 

Overall  plan 

Step  2 

Content  definition 

Step  3 

Test  specifications 

Step  4 

Item  development 

Step  5 

Test  design  and  assembly 

Step  6 

Test  production 

Step  7 

Test  administration 

Step  8 

Scoring  test  responses 

Step  9 

Passing  scores 

Step  10 

Reporting  test  results 

Step  11 

Item  banking 

Step  12 

Test  technical  report 
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c. 


USACE  51C/1102  PROFICIENCY  GUIDE  TASKS 


USACE 

Task  1-1 
Task  1-2 

Task  1-3 

Task  1-4 

Task  1-5 

Task  1-6 


Explain  the  Mission  of  the  U.S.  Army  Corps  of  Engineers 
Explain  the  Mission  and  Organizational  Structure  of  the 
National  Contracting  Organization  (NCO) 

Explain  the  USACE  Project  Management  Business  Process 
(PMBP) 

Explain  the  Differences  between  Civil  and  Military 
Construction  Contracting 

Correctly  Apply  Automated  Information  Systems  (AIS)  Used 
in  USACE  to  Accomplish  Construction  Contracting 
Explain  Types  of  Funding  Used  For  Construction  Contracts 


CONTRACTING  PRE-AWARD  TASKS  FOR  CONSTRUCTION 
CONTRACTING 

Task  2-1  Develop  an  Acquisition  Plan  for  a  Construction  Contract 
Task  2-2  The  Role  of  Small  Business  in  Contingency/Emergency 
Contracting 

Task  2-3  Develop  a  Source  Selection  Plan 

Task  2-4  Develop  a  Presolicitation  Announcement  for  Construction 
Task  2-5  Explain  How  the  Construction  Contract  Format  Differs  From 
the  Uniform  Contract  Format  Used  for  Other  Types  of 
Solicitations 

Task  2-6  Review  Construction  Plans  and  Specifications 
Task  2-7  Obtain  Presolicitation  Clearances 
Task  2-8  Develop  a  Construction  Solicitation 
Task  2-9  Conduct  Peer  Review 
Task  2-10  Issue  Solicitation  -  On  the  Street 
Task  2-11  Receive  Proposals  and  Determine  Responsiveness 
Task  2-12  Conduct  Source  Selection  Process 
Task  2-13  Explain  Contract  Funding  Process 
Task  2-14  Perform  Construction  Pre-Award  and  Award  Functions 
Task  2-15  Process  Pre  and  Post  Award  Protests 
Task  2-16  Procurement  of  Construction  Phase  Support  Services 
Task  2-17  Provide  Contingency  Contracting  Support  in  CONUS  to  a 
Federal  Emergency  Management  Agency  (FEMA) 
Contingency  Response  Team  (CRT) 


CONTRACTING  POST  AWARD  TASKS  FOR  CONSTRUCTION 
CONTRACTS 

Task  3-1  Notification  and  Debriefing  of  Unsuccessful  Offerors 
Task  3-2  Conduct  Post  Award  Functions  up  to  Notice  to  Proceed  (NTP) 
Task  3-3  Explain  the  Roles  and  Responsibilities  of  the  Construction 
Contract  Administration  Team 
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Task  3-4  Explain  the  Construction  Contract  Administration  Process  in 
USACE 

Task  3-5  Perform  Construction  Contract  Administration 
Task  3-6  Explain  Payments  Under  Construction  Contracts 
Task  3-7  Explain  Acceleration  and  Expediting  of  Construction  Contracts 
Task  3-8  Explain  the  Construction  Contract  Modification  Process 
Task  3-9  Establish  a  Profit  Objective  Using  the  Alternate  Weighted 
Guidelines  Method 

Task  3-10  Obtain  and  Use  a  Defense  Contract  Audit  Agency  (DCAA) 
Audit 

Task  3-11  Process  Claims 

Task  3-12  Perform  Construction  Contract  Closeout 
Task  3-13  Explain  the  Authority  for  and  Types  of  Contract  Terminations 
Task  3-14  Explain  Recurring  Issues  in  Construction  Contracts 
Task  3-15  Coordinate  with,  Prepare  for  and  Respond  to  Audits  and 
Inspections  by  Oversight  Agencies 

Task  3-16  Explain  the  Types  and  Battle  Rhythm  of  Datacalls  and 
Recurring  Reports 

ARCHITECT-ENGINEER  CONTRACTING 

Task  4-1  Explain  the  Authority  for  Architect-Engineer  (A-E) 
Contracting 

Task  4-2  Explain  the  Architect-Engineer  (A-E)  Selection  Process 
Task  4-3  Develop  an  Architect-Engineer  (A-E)  Solicitation/Request  for 
Price  Proposal 

Task  4-4  Negotiate  and  Award  and  Architect-Engineer  (A-E)  Contract 
Task  4-5  Execute  the  Architect-Engineer  (A-E)  Task  Order  Process 
Task  4-6  Execute  Post  Award  Actions  under  Architect-Engineer  (A-E) 
Contracts 
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D.  SAMPLE  TEST  ITEMS 


The  following  ten  items  are  examples  the  research  team  developed  for  the 
Proficiency  Assessment  Tool.  The  correct  selection  is  in  bold  font. 

1.  To  provide  vital  public  engineering  services  in  peace  and  war  to  strengthen  our 
Nation’s  security,  energize  the  economy  and  reduce  risks  from  disasters  is: 

A.  The  USACE  Mission 

B.  The  USACE  Vision 

C.  A  goal  of  USACE 

D.  A  USACE  objective 

Item  developed  for  Task  1—1:  Explain  the  Mission  of  the  U.S.  Army  Corps  of  Engineers 

2.  The _ is  responsible  for  oversight  and  technical  execution  of  the 

contracting  mission  in  USACE. 

A.  Head  of  Contracting  Activity 

B.  Regional  Contracting  Chief 

C.  Director  of  the  National  Contracting  Organization 

D.  Regional  PARC 

Item  developed  for  Task  1-2:  Explain  the  Mission  and  Organizational  Structure  of  the 
National  Contracting  Organization  (NCO) 

3.  Identify  the  primary  automated  information  system  (AIS)  used  in  the  development 
and  execution  of  a  construction  contract: 

A.  Corps  of  Engineers  Financial  Management  System  (CEFMS) 

B.  Standard  Procurement  System  (SPS) 

C.  Project  Management  Information  System  (P2) 

D.  Federal  Procurement  Data  System  (FPDS) 

Item  developed  for  Task  1-5:  Correctly  Apply  Automated  Information  Systems  (AIS) 
Used  in  USACE  to  Accomplish  Construction  Contracting 

4.  A  complete  requirements  package  is  accepted  by  contracting.  Assuming  no 
discussions,  the  PALT  for  a  large  Multiple  Award  Task  Order  Contract  using  LPTA 
should  take: 

A.  32  days 

B.  45  days 

C.  90  days 

D.  120  days 

Item  developed  for  Task  2—1:  Develop  an  Acquisition  Plan  for  a  Construction  Contract 
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5.  The  decision  regarding  the  designation  of  the  Source  Selection  Authority  must  be 

supported  by  a  recommendation  from  the _ . 

A.  District  or  Center  Contracting  Chief 

B.  Division  or  District  Commander 

C.  Principle  Assistant  Responsible  for  Contracting  (PARC) 

D.  Source  Selection 

Item  developed  for  Task  2-3:  Develop  a  Source  Selection  Plan 

6.  The  SI-CCCB  and  UFGS  WG  agreed  to  a  14  character  designation  of  MF  2004 
section  numbers  in  the  format  “NN  NN  NN.NN  NN”  where  N’s  are  numbers.  What  is 
the  subject  of  section  “01  00  00”? 

A.  Concrete 

B.  Existing  Conditions 

C.  General  Requirements 

D.  Procurement  and  Contracting  Requirements 

Item  developed  for  Task  2-5:  Explain  How  the  Construction  Contract  Format  Differs 
from  the  Uniform  Contract  Format  Used  for  Other  Types  of  Solicitations 

7.  Using  the  USACE  format,  identify  the  section  labeled  “Technical  Provisions” 

A.  00010 

B.  00100 

C.  00600C 

D.  16999 

Item  developed  for  Task  2-5:  Explain  How  the  Construction  Contract  Format  Differs 
from  the  Uniform  Contract  Format  Used  for  Other  Types  of  Solicitations 

8.  If  a  court  reporter  is  used  during  a  Pre-proposal  conference,  who  is  responsible  for 
procuring  the  services  of  the  court  reporter? 

A.  Contract  Specialist 

B.  Contracting  Officer 

C.  Project  Manager 

D.  The  assigned  Primary  Contractor 

Item  developed  for  Task  2-10:  Issue  Solicitation  -  On  The  Street 

9.  ACOs  may  be  warranted  to  make  contract  changes  under  applicable  clauses  for 

actions  not  exceeding  an  absolute  value  of  _ _ . 

A.  $100,000 

B.  $250,000 

C.  $500,000 

D.  $1,000,000 

Item  developed  for  Task  3-3:  Explain  Roles  and  Responsibilities  of  the  Construction 
Contract  Administration  Team 
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10.  Which  of  the  following  is  NOT  a  proper  Alternative  Dispute  Resolution  technique 
within  the  Corps  of  Engineers? 

A.  Mediation 

B.  Mini-trial 

C.  Non-binding  arbitration 

D.  Binding  arbitration 

Item  developed  for  Task  3-11:  Process  Claims 
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